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EXECUTIVE  SUMMARY 


This  thesis  describes  the  development  of  an  initial  Human  Readiness  Level  (HRL) 
framework,  designed  to  complement  the  existing  Technology  Readiness  Level  (TRL) 
currently  being  used  within  the  Integrated  Defense  Acquisition,  Technology,  and 
Logistics  (IDAT&L)  Life  Cycle  Management  System.  Technology  maturity  assessment 
tools,  such  as  the  TRL,  serve  as  systematic  measurement  systems  that  are  used  as  entry 
and  exit  criteria  for  transitioning  milestones  and  are  integral  components  to  program  risk 
management  structures.  Yet,  the  TRL  has  proven  incapable  of  consistently  capturing  the 
human-related  aspects  of  technology  development  and  their  association  with  technology 
readiness. 

The  proposed  HRL  framework  was  developed  with  the  aim  of  adding  clarity  to 
technical  readiness  assessments  by  emphasizing  the  socio-technical  attributes  of  system 
development.  Specifically,  the  HRL  aims  to  reduce  technology  risks  related  to  the  human 
element  by  ensuring  the  adequate  incorporation  of  Human  Systems  Integration  (HSI) 
during  technology  maturity  evaluations  and  by  explicitly  linking  technology  development 
to  the  effective  design  and  specification  of  human-centered  systems  in  Defense 
Acquisition.  The  HRL  accomplishes  this  by  providing  a  time  and  milestone-sensitive 
roadmap  of  activities  that:  a)  detail  critical  organizational  milestones  (that  ensure 
functional  commitment  to  HSI);  and  b)  define  HSI  domain-specific  data  collection  and 
analysis,  and  a  clear  process  for  their  management.  The  primary  measure  of  HSI  risk  and 
maturity  level  is  based  on  the  execution  and  outcome  of  those  milestone-sensitive 
management  and  analytical  activities  that  have  been  designated  in  each  progressive  HRL. 

To  evaluate  the  utility  of  the  initial  HRLs,  two  research  efforts  were  carried  out. 
In  the  first,  an  evaluation  questionnaire  was  designed  to  gain  feedback  as  to  the  overall 
worth  and  potential  usefulness  of  the  HRL  framework.  The  questionnaire  was  given  to  43 
subject  matter  experts  (SMEs)  in  the  fields  of  HSI  and  Defense  Acquisition.  A  total  of  15 
responses  were  obtained.  Results  of  this  evaluation  indicated  agreement  among  SMEs 
that  the  first  iteration  of  the  HRL  concept,  once  fully  developed,  will  serve  as  a  valuable 
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HSI  planning  tool  and  complementary  (HSI-specific)  metric  in  program  risk  management 
structures.  In  order  to  improve  the  framework,  many  SMEs  recommended  expanding  the 
HRL’s  list  of  activities  to  account  for  a  more  diverse  variety  of  programs  that  exist  in 
acquisition. 

The  second  research  effort  was  a  case  study  regarding  the  usefulness  of  the  HRL 
framework  as  applied  to  a  specific  acquisition  program.  Specifically,  the  HRL  was 
evaluated  in  its  ability  to  effectively  contribute  to  the  creation  and  sustainment  of  the 
acquisition  program’s  HSI  Plan  (HSIP)  for  the  Ground  Control  Station  Modernization 
(GMOD).  Results  of  this  SME  evaluation  suggested  a  general  agreement  that  the  HRL 
framework  could  serve  as  a  beneficial  tool  in  the  development  of  an  acquisition 
program’s  comprehensive  HSIP.  Consistent  with  the  findings  from  the  first  research 
effort,  the  SME  did  not  feel  the  HRL  framework  contained  all  of  the  necessary  HSI 
activities  to  account  for  all  program  needs.  The  feedback  and  lessons  learned  from  both 
studies  were  documented  and  will  serve  to  form  a  basis  of  advocacy  for  future  HRL 
development  efforts. 

Further  development  and  evaluation  of  the  HRL  concept  is  required.  However, 
the  initial  framework  presented  in  this  thesis  takes  that  first  step  towards  providing 
acquisition  professionals  a  comprehensive  guide  that  ensures  human-centric  priorities  are 
addressed  throughout  all  phases  and  milestones  of  Defense  Acquisition. 
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I.  INTRODUCTION 


A.  PURPOSE 

This  thesis  develops  and  documents  an  executable  process  description  of  the 
Human  Readiness  Level  (HRL)  to  complement  the  existing  Technology  Readiness  Level 
(TRL)  measurement  process  currently  being  used  within  the  Integrated  Defense 
Acquisition,  Technology,  and  Logistics  (IDAT&L)  Life  Cycle  Management  System. 
Technology  maturity  measurement  tools,  such  as  the  TRL,  serve  as  systematic 
measurement  systems  that  describe  the  maturity  of  a  particular  technology  and  allow 
consistent  comparison  of  maturity  to  be  made  between  different  types  of  technologies 
(Saucer,  Verma,  Ramirez-Marquez,  &  Gove,  2006).  Throughout  Department  of  Defense 
(DoD)  acquisition  programs,  TRLs  are  used  as  entry  and  exit  criteria  for  transitioning 
milestones  and  are  integral  components  to  program  risk  management  structures.  The 
proposed  HRL  measurement  framework  will  add  clarity  to  technical  readiness 
assessments  by  emphasizing  the  socio-technical  attributes  of  systems.  The  HRL  aims  to 
reduce  technology  risks  related  to  the  human  element  by  ensuring  the  inclusion  of  Human 
Systems  Integration  (HSI)  considerations  during  technology  maturity  evaluations.  By 
capturing  human-related  components  of  evolving  technologies,  the  acquisition  and 
Science  &  Technology  (S&T)  communities  will  decrease  operational  and  development 
risks  linked  to  insufficient  HSI. 

B.  OVERVIEW 

The  DoD  relies  heavily  on  the  technological  superiority  of  its  weapon  systems 
and  armed  forces  to  protect  U.S.  interests.  Although  the  United  States  has  produced  some 
of  the  finest  modem  weapon  systems  in  the  world,  the  Government  Accountability  Office 
(GAO)  points  out  that  its  acquisition  programs  often  incur  cost  overruns,  schedule  delays, 
and  performance  shortfalls  that  weaken  DoD’s  buying  power  (GAO,  2006).  This  ongoing 
challenge  is  due  in  part  to  DoD’s  difficulty  in  managing  technology  maturity.  In 
numerous  reports  to  Congressional  Committees,  the  GAO  has  addressed  the  problems 
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with  proceeding  in  system  development  with  immature  technologies  and  has  detailed 
how  best  practices  offer  improvements  to  the  way  the  DoD  incorporates  new  technology 
into  weapon  system  programs.  According  to  a  2001  GAO  report,  “organizations  that  use 
best  practices  recognize  that  delaying  the  resolution  of  technology  problems  until 
product  development — analogous  to  the  Engineering  and  Manufacturing  Development 
phase — can  result  in  at  least  a  ten-fold  cost  increase;  delaying  the  resolution  until  after 
the  start  of  production  could  increase  costs  by  a  hundred-fold'’  (GAO  JSF,  2001,  p.  1-2). 
The  GAO  went  on  to  illustrate  this  point  by  providing  the  following  comparison  between 
the  Joint  Direct  Attack  Munition  (JDAM)  and  the  Comanche  helicopter  programs. 

For  example,  the  Joint  Direct  Attack  Munition  used  modified  variants  of 
proven  components  for  guidance  and  global  positioning.  It  also  used 
mature,  existing  components  from  other  proven  manufacturing  processes 
for  its  own  system  for  controlling  tail  fin  movements.  The  munition  was 
touted  for  its  performance  in  Kosovo  and  was  purchased  for  less  than  half 
of  its  expected  unit  cost.  However,  the  Comanche  helicopter  program 
began  with  critical  technologies  such  as  the  engine,  rotor,  and  integrated 
avionics  at  Technology  Readiness  Levels  (TRLs)  of  5  or  below  [see  later 
for  detailed  discussion  of  TRLs] .  That  program  has  seen  101  percent  cost 
growth  and  120-percent  schedule  slippage  as  a  result  of  these  low 
maturity  levels  and  other  factors.  (GAO  JSF,  2001,  p.  7) 

To  enhance  the  ability  of  programs  to  select  mature  technologies  for  inclusion  in 
their  programs,  the  GAO  recommended  the  use  of  TRLs  (Graettinger  et  ah,  2002).  The 
first  published  description  of  technology  readiness  can  be  found  in  a  1989  paper  titled, 
“The  NASA  Technology  Push  towards  Future  Space  Mission  Systems .”  In  that  paper, 
Stanley  Sadin  and  his  co-authors  presented  a  seven-level  readiness  metric  that  assessed 
the  risk  associated  with  technology  development.  In  the  years  that  followed,  the  metric 
evolved  into  the  nine  levels  that  exist  today.  The  TRL  describes  the  maturity  of 
technology  elements  with  levels  that  begin  in  the  earliest  stages  of  scientific  investigation 
(Level  1)  to  the  successful  use  in  the  system  (Level  9).  Organizations,  such  as  the  Air 
Force  Research  Laboratory  (AFRL)  promotes  TRLs  as  a  means  of  evaluating  the 
readiness  of  technologies  to  be  incorporated  into  a  weapon  or  other  type  of  military 
system  (Graettinger  et  ah,  2002). 
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Despite  the  practical  utility  of  using  TRLs  to  measure  technology  readiness,  they 
fail  to  account  for  the  critical  human  element.  By  inadequately  considering  the  human 
capacity  or  requirements  as  part  of  the  system,  the  ability  to  enhance  human/system 
performance,  maximize  operational  effectiveness,  and  reduce  life  cycle  cost  (LCC)  is 
compromised  (711  HPW/HPO,  2009).  Therefore,  additional  methodologies  are  needed  to 
ensure  that  human-centric  priorities  are  implemented  throughout  all  phases  and 
milestones  of  defense  acquisition.  To  this  end,  it  is  the  goal  of  this  thesis  to  provide  the 
acquisition  and  S&T  communities  a  human-centric  approach  to  assessing  technology 
maturity  in  order  to  avoid  operational  and  development  risks  associated  with  insufficient 
consideration  of  the  human  element. 

C.  NEED  FOR  HUMAN  SYSTEMS  INTEGRATION 

The  level  of  technology  that  the  U.S.  provides  to  its  armed  forces  is  simply 
unparalleled.  However,  even  with  sophisticated  technologies,  optimized  total  systems 
performance  still  depends  on  the  warfighter’s  ability  to  use  the  technology  fully  and 
effectively  (AFHSIO,  2009).  HSI  serves  as  the  critical  link  that  infuses  human  roles, 
structures,  processes,  and  constraints  into  the  design  of  systems  to  achieve  total  systems 
performance.  By  integrating  HSI  strategies  throughout  a  system’s  life  cycle,  programs  are 
better  able  to  identify,  track  and  resolve  human-related  issues  and  ensure  a  balanced 
development  of  both  human  and  technological  capabilities  (MoD  HFI  DTC,  2008).  This 
is  accomplished  by  incorporating  functional  areas,  referred  to  as  domains.  These  domains 
are  identified  in  the  DoD  instruction  on  the  operation  of  the  Defense  Acquisition  System 
(DoDI  5000.02)  and  are  illustrated  in  Table  1: 
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Table  1. 


HSI  Domains  and  Definitions  (After  711  HPW/HPO,  2009) 


HSI  Domain 

Definition 

Manpower 

The  number  and  mix  of  personnel  (military,  civilian,  and 
contractor)  authorized  and  available  to  train,  operate, 
maintain,  and  support  each  system  acquisition. 

Personnel 

The  human  aptitudes,  skills,  knowledge,  experience 
levels,  and  abilities  required  to  operate,  maintain,  and 
support  the  system  at  the  time  it  is  fielded  and 
throughout  its  life  cycle. 

Training 

The  instruction  and  resources  required  to  provide 
personnel  with  requisite  knowledge,  skills,  and  abilities 
to  properly  operate,  maintain,  and  support  the  system. 

Human  Factors  Engineering 

The  comprehensive  integration  of  human  capabilities 
and  limitations  (cognitive,  physical,  sensory,  and  team 
dynamic)  into  system  design,  development,  modification 
and  evaluation  to  optimize  human-machine  performance 
for  both  operation  and  maintenance  of  a  system.  Human 
factors  engineering  designs  systems  that  require  minimal 
manpower,  provide  effective  training,  can  be  operated 
and  maintained  by  users;  and  are  suitable  and 
survivable. 

Environment 

The  factors  concerning  water,  air,  and  land  and  the 
interrelationships  which  exist  among  and  between  water, 
air,  and  land  and  all  living  things. 

Safety 

The  design  and  operational  characteristics  that  minimize 
the  possibilities  for  accidents  or  mishaps  to  operators 
which  threaten  the  survival  of  the  system. 

Occupational  Health 

The  design  features  that  minimize  risk  of  injury,  acute 
and/or  chronic  illness,  or  disability,  and/or  reduced  job 
performance  of  personnel  who  operate,  maintain,  or 
support  the  system. 

Survivability 

The  characteristics  of  a  system  that  reduce  risk  of 
fratricide,  detection,  and  the  probability  of  being 
attacked;  and  that  enable  the  crew  to  withstand  man¬ 
made  or  natural  hostile  environments  without  aborting 
the  mission  or  suffering  acute  and/or  chronic  illness, 
disability,  or  death. 

Habitability 


Factors  of  living  and  working  conditions  that  is 
necessary  to  sustain  the  morale,  safety,  health,  and 
comfort  of  the  user  population  which  contribute  directly 
to  personnel  effectiveness  and  mission  accomplishment, 
and  often  preclude  recruitment  and  retention  problems. 


Although  HSI  is  a  fundamental  component  of  a  total  systems  approach,  the 
successful  integration  of  HSI  into  systems  engineering  and  acquisition  life  cycles 
continues  to  be  a  challenge.  The  Air  Force  71 1th  Human  Performance  Wing  points  to  the 
following  barriers  that  often  exist  in  program  development: 

•  Lack  of  alignment  of  processes  between  the  traditional  practice  of  HSI  and 
the  human-based  technical  domains; 

•  Lack  of  alignment  between  human  technical  areas  and  the  systems 
engineering  and  program  management  practices  of  DoD  acquisition;  and 

•  Inability  to  communicate,  which  causes  an  inability  to  share  data  or 
information  throughout  the  acquisition  phases  and  beyond  (711 
HPW/HPO,  2008). 

Therefore,  recognizing  this  need  for  HSI  involvement  and  a  sustainable  human 
approach  to  system  acquisition,  there  has  been  an  international  emergence  of  what  is 
being  termed,  the  Human  View.  The  Human  View  was  originally  created  to  be  used  as  a 
complementary  element  to  the  Operational,  System  and  Technical  Standards  Views  of  the 
Ministry  of  Defence  Architectural  Framework  (MODAF).  The  Human  View  aims  to 
clarify  the  role  of  Human  Factors  Integration  (HFI;  the  United  Kingdom  (UK)  equivalent 
of  HSI)  when  creating  Enterprise  Architectures  in  support  of  acquisition  (MoD  HFI  DTC, 
2008). 

Architectures  such  as  MODAF  or  the  US-created  DoD  Architectural  Framework 
(DoDAF)  produce  common  systems  engineering  (SE)  approaches  to  development, 
presentation,  and  integration  of  current  and  future  systems.  Within  DoD,  architectures 
are  created  for  a  number  of  reasons.  From  a  practical  perspective,  experience  has  shown 

that  the  management  of  large  organizations  employing  sophisticated  systems  in  pursuit  of 
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joint  missions  demands  an  organized,  repeatable  method  for  evaluating  investments  and 
investment  alternatives,  as  well  as  the  ability  to  effectively  implement  organizational 
change,  create  new  systems,  and  deploy  new  technologies  (DoD,  2007).  Towards  this 
end,  the  Human  View  connects  HSI  to  DoDAF  by  supporting  system  design  and 
development  that  effectively  and  affordably  integrates  human  capabilities  and  limitations. 

Although  the  Human  View  has  been  tailored  to  the  major  characteristics  of 
architectural  models,  HSI  practitioners  working  within  Defense  Acquisition  have 
observed  the  potential  for  extended  applications.  An  example  of  this  extended 
application,  which  serves  as  the  focus  of  this  thesis,  was  conceived  by  Dr.  Hector  Acosta 
of  the  Air  Force  711th  Human  Performance  Wing  (a  co-advisor  of  this  thesis).  After 
recognizing  the  potential  for  using  Human  View  technical  details  and  attributes  to 
structure  HSI  initiatives  within  technology  readiness  detenninations,  Dr.  Acosta  created 
the  concept  of  the  HRL.  This  thesis  will  build  upon  the  work  of  Dr.  Acosta  by  developing 
the  HRL  concept  to  fonn  a  defined  and  executable  process  description.  This  will  allow 
HRLs  to  immediately  become  applicable  to  the  acquisition  and  S&T  communities.  HRLs 
will  facilitate  the  unification  and  integration  of  HSI  processes  within  technology 
evaluations,  to  include  formal  and  informal  technology  maturity  assessments.  Ultimately, 
HRLs  will  provide  DoD  acquisition  programs  the  ability  to  reduce  technology  risks 
related  to  the  human  element  and  will  help  to  maximize  both  return  on  investment  (ROI) 
and  system  performance. 

D.  SCOPE 

This  thesis  focused  on  defining  Human  Readiness  Levels  for  use  in  technology 
development  life  cycles,  using  the  detailed  elements  of  the  Human  View.  The  tasks 
comprising  this  thesis  included  the  following: 

•  Conduct  in-depth  analysis  of  current  DoD  technology  development 
processes  and  maturity  standards  (i.e.,  Technology  Readiness  Assessment 
(TRA),  TRL,  etc.)  adhered  to  in  the  Defense  Acquisition  System  and 
document  shortfalls  in  the  consideration  of  human  aspects  of  systems. 
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•  Synthesize  the  technical  details  of  the  Human  View  and  its  relation  to 
DoDAF  and  explicitly  develop  its  linkage  to  the  concept  of  an  HRL. 

•  Analyze  the  Human  View/HRL  amalgamation’s  validity  and  usability  in 
acquisition  and  S&T  environments  and  begin  the  effort  towards 
elaborating  the  HRL  concept  throughout  the  lifecycle  of  a  system. 

E.  THESIS  ORGANIZATION 

A  brief  literature  review  of  the  significance  and  history  of  technology  maturity  is 
discussed  in  Chapter  II.  This  review  encompasses  the  nine  Technology  Readiness  Levels 
used  by  government  and  industry  to  measure  technology  maturity  and  program  risk,  and 
highlights  the  inability  of  the  readiness  levels  to  adequately  capture  human-specific 
elements  of  evolving  technologies.  To  approach  the  issue,  this  thesis  develops  and 
defines  the  complimentary  concept  of  the  Human  Readiness  Level.  Because  the  HRL 
employs  the  technical  details  of  the  Human  View  to  structure  HSI  initiatives  specifically 
within  technology  maturity  assessments,  the  history  and  utilization  of  the  Human  View 
also  is  discussed  in  Chapter  II.  The  subsequent  chapters  are  devoted  to  introducing  and 
analyzing  the  HRL’s  value,  validity,  and  usability  in  acquisition  and  S&T  environments. 
Chapter  III  details  the  development  process  behind  the  HRL  and  presents  the  first 
iteration  of  the  HRL  concept.  Chapter  IV  discusses  the  initial  evaluation  of  the  HRL 
concept.  This  included  Subject  Matter  Expert  (SME)  feedback  that  focused  on  accuracy, 
ease  of  use,  and  completeness  of  the  HRL  framework.  Chapter  V  describes  the 
succeeding  validation  of  the  HRL  model  being  practically  applied  in  a  brief  case  study. 
Chapter  VI  summarizes  this  thesis  effort  and  discusses  future  implications  for  HRL  use  in 
Defense  Acquisition. 
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II.  LITERATURE  REVIEW 


A.  TECHNOLOGY 

1.  Technology  Defined 

The  National  Aeronautics  and  Space  Administration  (NASA)  Technology  Plan 
defines  technology  as  the  practical  application  of  knowledge  to  create  the  capability  to  do 
something  entirely  new  or  in  an  entirely  new  way.  This  can  be  contrasted  to  scientific 
research,  which  encompasses  the  discovery  of  new  knowledge  from  which  new 
technology  is  derived,  and  engineering  that  uses  technology  derived  from  this  knowledge 
to  solve  specific  technical  problems  (Bilbro,  2007).  Similarly,  Dictionary. Com  defines 
technology  as  follows: 

The  branch  of  knowledge  that  deals  with  the  creation  and  use  of  technical 
means  and  their  interrelation  with  life,  society,  and  the  environment, 
drawing  upon  such  subjects  as  industrial  arts,  engineering,  applied 
science,  and  pure  science;  a  technological  process,  invention,  method,  or 
the  like;  the  sum  of  the  ways  in  which  social  groups  provide  themselves 
with  the  material  objects  of  their  civilization.  (Technology,  n.d.) 

Science  and  technology  are  terms  that  are  often  seen  as  one  and  the  same. 
However,  William  Nolte  (2008)  points  out  in  his  book,  “Did  I  Ever  Tell  You  about  the 
Whale?  Or  Measuring  Technology  Maturity,  ”  that  science  and  technology  are  related, 
but  not  identical.  He  states  that  science  is  the  search  for  new  knowledge  for  knowledge’s 
sake,  while  technology  takes  that  knowledge  of  science  and  makes  it  do  something  useful 
for  society.  Therefore,  when  scientific  knowledge  is  put  to  use,  we  have  technology.  The 
form  in  which  technology  takes  shape  can  be  seen  in  hardware  (i.e.,  cell  phone, 
computer,  aircraft,  etc.)  or  in  non-material  technologies  (i.e.,  software,  new  practices  or 
procedures;  Nolte,  2008).  This  understanding  of  what  technology  is  and  what  it  may 
encompass  will  serve  as  the  contextual  foundation  for  the  remaining  discussions  in  this 
chapter’s  literature  review. 
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The  concept  of  technology  maturity  and  how  it  is  currently  measured  in 
government  and  industry  will  first  be  examined  below,  followed  by  the  limitations 
inherent  in  such  measurements.  After  which,  the  discussion  focuses  on  the  need  for  a 
human  approach  to  technology  readiness  and  describes  the  resulting  technological 
immaturity  that  occurs  when  the  human  is  not  adequately  incorporated.  This  chapter  ends 
with  the  emergence  of  the  Human  View  and  details  the  ability  of  the  Human  View  to 
integrate  and  shape  human-specific  elements  into  the  design  and  implementation  of 
technology. 

2.  Technology  Maturity 

With  an  understanding  of  what  technology  is  and  what  it  may  encompass,  it  is 
appropriate  to  now  examine  what  technology  maturity  might  mean  and  how  it  is  currently 
measured.  In  Dr.  Tom  Cruse’s  2006  report,  ‘ Multi-Dimensional  Assessment  of 
Technology  Maturity ’  he  points  out  that  the  tenn,  maturity  implies  aging,  or  growth.  With 
technology  growth,  Cruse  states  that  the  technology  itself  does  not  necessarily  change, 
but  rather  our  understanding  of  the  technology  changes.  He  explains  that  as  our 
understanding  improves  or  grows,  the  technology’s  usefulness  should  improve  as  well. 
Therefore,  the  more  we  leam  about  a  technology,  the  better  we  can  apply  it  to  meet  our 
needs.  For  the  most  part,  the  customer  or  end  user  of  a  technology  intuitively  knows  what 
it  means  for  a  technology  to  mature  because  they  will  often  wait  for  a  new,  immature 
product  to  “get  the  bugs  out”  before  purchasing.  The  expectation  for  a  technology  to 
become  more  mature,  more  capable  of  meeting  needs  and  requirements  after  it  “grows 
up”  is  the  basic  premise  of  technological  maturity  (Nolte,  2008). 

Within  the  DoD,  the  evolution  and  management  of  technology  maturity  continues 
to  be  a  subject  of  study  and  emphasis.  In  their  work  on  best  business  practices  in  the  last 
two  decades,  the  GAO  studied  a  number  of  leading  commercial  firms  to  determine  key 
factors  in  successful  product  development.  They  reported  that  one  such  key  factor  is 
maturing  a  new  technology  far  enough  to  get  it  into  the  right  size,  weight,  and 
configuration  needed  for  the  intended  product.  After  this  is  demonstrated,  the  technology 
is  said  to  be  at  an  acceptable  level  for  product  development  (Graettinger  et  al.,  2002). 
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According  to  the  GAO,  "...  no  element  is  more  important  than  having  technology 
advanced  enough  to  meet  requirements,  but  also  mature  enough  to  be  predictably 
managed  and  available  at  the  start  of  the  product  development  cycle.  ”  They  further 
stated  that,  “ maturing  new  technology  before  it  is  included  on  a  product  is  perhaps  the 
most  important  determinant  of  the  success  of  the  eventual  product — or  weapon  system  ” 
(GAO,  1999,  p.  12). 

To  subjectively  quantify  the  maturity  of  certain  technologies  and  to  ensure  that 
new  technologies  are  vigorously  pursued  and  successfully  moved  into  program 
development,  the  adoption  of  a  disciplined  and  knowledge-based  method  is  often 
recommended  (GAO,  1999).  The  beginnings  of  such  a  method  can  be  traced  back  as 
early  as  1969,  where  NASA  articulated  the  status  of  new  technology  planned  for  use  in 
future  space  systems.  The  examples  used  were  between  the  then-established  practice  of 
the  “flight  readiness  review,”  and  a  new  idea  through  which  the  level  of  maturity  of  new 
technologies  could  be  assessed — the  “technology  readiness  review.” 

Although  NASA  used  the  emerging  concept  of  technology  readiness  for  some 
time  after,  the  actual  readiness  level  scale  was  not  devised  and  published  until  1989.  That 
year,  in  an  effort  to  develop  a  “systems-technology  model,”  Mr.  Stan  Sadin  of  the  Office 
of  Aeronautics  and  Space  Technology  codified  seven  levels  of  technology  readiness.  The 
seven  readiness  levels  showed  how  the  more  traditional  breakdown  of  research  and 
development  were  incorporated  into  four  loosely  defined  categories  (Nolte,  2008):  basic 
research;  feasibility;  development;  and  demonstration 

These  readiness  levels  marked  a  significant  change  in  emphasis  on  the  part  of 
NASA  where  technology  had  previously  been  viewed  as  merely  having  a  supporting  role. 
This  change  in  role  for  technology  was  the  result  of  a  revision  in  the  National  Space 
Policy  stating  that  NASA’s  technology  program  "...  shares  the  mantle  of  responsibility 
for  shaping  the  Agency's  future...  ”  The  new  emphasis  on  technology’s  responsibility  was 
cause  for  a  reassessment  of  how  technologies  were  developed  and  infused,  with  a  goal  of 
approaching  technology  development  and  infusion  in  a  much  more  systematic  way  to 
increase  the  likelihood  of  success  (JB  International,  n.d.).  As  Sadin,  Povinelli,  and  Rosen 

(1989)  point  out,  when  historical  records  are  analyzed,  it  can  be  demonstrated  that  the 
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difference  between  success  and  failure  is  traceable  to  the  adequacy,  or  depth,  of  the 
advanced  research  and  technology  program  pursuing  technology  readiness. 

Within  DoD,  the  concept  of  technology  readiness  was  not  entirely  embraced  until 
1999.  That  year,  the  GAO  conducted  an  assessment  of  how  best  practices  offer 
improvements  to  the  way  DoD  incorporates  new  technologies  into  Defense  Acquisition 
programs.  The  GAO  concluded  that  “the  experiences  of  DoD,  and  the  commercial 
technology  development  cases  GAO  reviewed,  indicate  that  demonstrating  a  high  level  of 
maturity  before  new  technologies  are  incorporated  into  product  development  programs 
puts  those  programs  in  a  better  position  to  succeed ”  (GAO,  1999,  p.  3).  Soon  after,  in  a 
2001  memorandum,  the  Deputy  Undersecretary  of  Defense  for  Science  and  Technology 
officially  endorsed  the  use  of  the  Technology  Readiness  Level  (TRL)  as  the  standard 
format  for  measuring  technology  maturity  in  new  major  programs  within  Defense 
Acquisition  (Graettinger  et  ah,  2002). 

TRLs  follow  a  scale  from  1  ( basic  research)  to  9  ( system  operation).  A 
technology  determined  to  be  at  TRL  1  is  at  the  lowest  level  of  technology  readiness, 
where  “scientific  research  begins  to  be  translated  into  applied  research  and  development” 
(GAO,  2006,  p.  56).  By  the  time  the  technology  has  reached  a  TRL  9,  the  technology  has 
progressed  through  formulation  of  an  initial  concept  for  application,  proof  of  concept, 
demonstration  in  a  laboratory  environment  and  realistic  environment,  and  integration  into 
a  system,  and  has  been  “system  qualified”  and  then  “system  proven.”  This  last  state  of 
development,  where  the  technology  is  operating  under  actual  mission  conditions,  is  TRL 
9  (Valerdi,  2004).  Table  2  provides  the  DoD’s  original  technology  readiness  levels  and 
descriptions  from  a  systems  perspective  with  supplemental  definitions  provided  in  Table 
3  (separate  matrices  for  DoD’s  hardware  and  software  TRLs  are  detailed  extensively  in 
Appendices  A  and  B). 
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Table  2. 


DoD  Technology  Readiness  Levels  (After  GAO,  2006) 


Technology  Readiness  Level 

Description 

1.  Basic  principles  observed  and  reported. 

Lowest  level  of  technology  readiness. 
Scientific  research  begins  to  be  translated 
into  applied  research  and  development. 
Examples  might  include  paper  studies  of  a 
technology’s  basic  properties. 

2.  Technology  concept  and/or  application 
formulated. 

Invention  begins.  Once  basic  principles  are 
observed,  practical  applications  can  be 
invented.  Applications  are  speculative  and 
there  may  be  no  proof  or  detailed  analysis 
to  support  the  assumption.  Examples  are 
still  limited  to  analytic  studies. 

3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept. 

Active  research  and  development  is 
initiated.  This  includes  analytical  studies 
and  laboratory  studies  to  physically 
validate  analytical  predictions  of  separate 
elements  of  the  technology.  Examples 
include  components  that  are  not  yet 
integrated  or  representative. 

4.  Component  and/or  breadboard. 
Validation  in  laboratory  environment. 

Basic  technological  components  are 
integrated  to  establish  that  they  will  work 
together.  This  is  relatively  “low  fidelity” 
compared  to  the  eventual  system. 
Examples  include  integration  of  “ad  hoc” 
hardware  in  a  laboratory. 

5.  Component  and/or  breadboard  validation 
in  a  relevant  environment. 

Fidelity  of  breadboard  technology 
increases  significantly.  The  basic 

technological  components  are  integrated 
with  reasonably  realistic  supporting 
elements  so  it  can  be  tested  in  a  simulated 
environment.  Examples  include  “high 
fidelity”  laboratory  integration  of 

components. 

6.  System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment. 

Representative  model  or  prototype  system, 
which  is  well  beyond  that  of  TRL  5,  is 
tested  in  a  relevant  environment. 
Represents  a  major  step  up  in  a 
technology’s  demonstrated  readiness. 
Examples  include  testing  a  prototype  in  a 
high-fidelity  laboratory  environment  or  in 
simulated  operational  environment. 
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7.  System  prototype  demonstration  in  an 
operational  environment. 

Prototype  near,  or  at,  planned  operational 
system.  Represents  a  major  step  up  from 
TRL  6,  requiring  demonstration  of  an 
actual  system  prototype  in  an  operational 
environment  such  as  in  an  aircraft,  vehicle, 
or  space.  Examples  include  testing  the 
prototype  in  a  test  bed  aircraft. 

8.  Actual  system  completed  and  qualified 
through  test  and  demonstration. 

Technology  has  been  proven  to  work  in  its 
final  form  and  under  expected  conditions. 
In  almost  all  cases,  this  TRL  represents  the 
end  of  true  system  development.  Examples 
include  developmental  test  and  evaluation 
of  the  system  in  its  intended  weapon 
system  to  determine  if  it  meets  design 
specifications. 

9.  Actual  system  proven  through  successful 
mission  operations. 

Actual  application  of  the  technology  in  its 
final  fonn  and  under  mission  conditions, 
such  as  those  encountered  in  operational 
test  and  evaluation.  Examples  include 
using  the  system  under  operational  mission 
conditions. 

Table  3.  Additional  Definitions  of  TRL  Descriptive  Terms  (From  DoD,  2009) 


Term 

Definition 

Breadboard 

Integrated  components  that  provide  a 
representation  of  a  system/subsystem  and 
that  can  be  used  to  determine  concept 
feasibility  and  to  develop  technical  data. 
Typically  configured  for  laboratory  use  to 
demonstrate  the  technical  principles  of 
immediate  interest.  May  resemble  final 
system/subsystem  in  function  only. 

High  Fidelity 

Addresses  form,  fit,  and  function.  A  high- 
fidelity  laboratory  environment  would 
involve  testing  with  equipment  that  can 
simulate  and  validate  all  system 
specifications  within  a  laboratory  setting. 

Low  Fidelity 

A  representative  of  the  component  or 
system  that  has  limited  ability  to  provide 
anything  but  first-order  information  about 
the  end  product.  Low-fidelity  assessments 
are  used  to  provide  trend  analysis. 

14 


Model 

A  functional  form  of  a  system,  generally 
reduced  in  scale,  near  or  at  operational 
specification.  Models  will  be  sufficiently 
hardened  to  allow  demonstration  of  the 
technical  and  operational  capabilities 
required  of  the  final  system. 

Operational  Environment 

Environment  that  addresses  all  the 
operational  requirements  and  specifications 
required  of  the  final  system  to  include 
platform/packaging. 

Prototype 

A  physical  or  virtual  model  used  to 
evaluate  the  technical  or  manufacturing 
feasibility  or  military  utility  of  a  particular 
technology  or  process,  concept,  end  item, 
or  system. 

Relevant  Environment 

Testing  environment  that  simulates  both 
the  most  important  and  most  stressing 
aspects  of  the  operational  environment. 

Simulated  Operational  Environment 

Either  (1)  a  real  environment  that  can 
simulate  all  the  operational  requirements 
and  specifications  required  of  the  final 
system  or  (2)  a  simulated  environment  that 
allows  for  testing  of  a  virtual  prototype. 
Used  in  either  case  to  determine  whether  a 
developmental  system  meets  the 

operational  requirements  and  specifications 
of  the  final  system. 

To  illustrate  how  technologies  progress  from  initial  concept  to  proven 
performance  within  the  readiness  levels,  the  following  hypothetical  example  about  an 
airborne  communications  radio  is  provided: 

First,  the  idea  for  a  new  radio  is  conceived.  The  idea  reaches  TRL  3  when 
analytical  studies  and  some  tests  of  the  technology ’s  elements,  such  as  a 
circuit,  back  it  up.  When  initial  hand-built  versions  of  all  of  the  radio ’s 
basic  elements  are  connected  and  tested  together,  the  radio  reaches  TRL 
5.  This  is  sometimes  referred  to  as  a  “ breadboard ”  article;  although  it 
may  function  like  a  radio,  it  does  not  look  like  one  because  the  individual 
parts  are  attached  to  plywood  and  hand-wired  together.  When  the 
technology  is  built  into  a  generic  model,  which  is  well  beyond  the 
breadboard  tested  in  TRL  5,  and  demonstrated  in  a  laboratory 
environment,  the  radio  reaches  TRL  6.  This  model  represents  the  last  level 
of  demonstration  before  the  radio  becomes  tailored  for  application  to  a 
specific  aircraft.  When  the  components  are  assembled  inside  a  case  that 
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resembles  the  final  radio  design  and  are  demonstrated  aboard  a  surrogate 
for  the  intended  aircraft,  the  radio  reaches  TRL  7.  TRL  8  is  reached  when 
the  radio  is  put  in  its  final  form,  installed  in  the  intended  aircraft’s 
cockpit,  and  tested  in  conjunction  with  the  other  aircraft  equipment  with 
which  it  must  interface.  TRL  9  is  achieved  when  the  radio  is  successfully 
operated  on  the  aircraft  through  several  test  missions.  (GAO,  1999,  p.  22- 
23) 

The  TRL  offers  many  benefits  to  the  S&T  and  acquisition  communities.  For 
example,  they  provide  a  snapshot  of  a  technology’s  maturity  at  a  given  instant  and  can  be 
used  to  establish  benchmarks  for  maturity  goals.  The  TRL  can  also  serve  as  a 
communication  device  in  those  cases  where  a  technology  developer  must  hand  off  a 
technology  to  another  organization  for  product  or  system  development.  TRLs  can  assist 
both  sides  in  understanding  exactly  what  each  is  required  to  do.  By  providing  a  common 
reference  point  for  the  technology  developer  and  user,  TRLs  assist  in  eliminating 
misunderstandings  and  ambiguities  in  the  technology  transition  process  (Nolte,  2008). 

B.  LIMITATIONS  OF  THE  TRL 

Despite  the  practical  utility  of  using  TRLs  to  measure  technology  readiness,  they 
do,  however,  have  significant  limitations.  For  instance,  author  Jim  Smith  (2005)  states 
that  TRL  definitions  often  combine  several  different  aspects  of  technology  readiness  into 
one  metric,  thus  “blurring”  the  different  contributors  to  readiness.  An  example  of  this  can 
be  seen  in  how  the  U.S.  Army  Communications  Electronics  Command  (CECOM)  defines 
a  TRL  7  in  their  draft  software  TRL  scale  (Graettinger  et  ah,  2002): 

Represents  a  major  step  up  from  TRL  6,  requiring  demonstration  of  an 
actual  system  prototype  in  an  operational  environment...  Algorithms  run 
on  the  processor  of  the  operational  system  and  are  integrated  with  actual 
external  entities.  Software  support  structure  is  in  place.  Software  releases 
are  in  distinct  versions.  Frequency  and  severity  of  software  deficiency 
reports  do  not  significantly  degrade  functionality  or  performance. 

This  description  combines  the  functionality  of  algorithms,  the  maintainability  of  a 
software  support  structure,  and  the  reliability  of  software  deficiency  reports  to  indicate  a 
specific  level  of  readiness  at  TRL  7.  The  problem,  however  is  that  the  manner  in  which 
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these  factors  are  combined  makes  it  difficult  to  understand  how  any  one  aspect  influences 
the  overall  readiness  of  the  technology  (Smith,  2005). 

Similarly,  William  Nolte  states: 

The  TRL  scale  measures  maturity  along  a  single  axis,  the  axis  of 
technology  capability  demonstration.  A  full  measure  of  technology 
maturity,  or  in  the  commercial  world,  product  maturity,  would  be  a  multi¬ 
dimensional  metric.  It’s  not  uncommon  to  find  references  to  12  or  more 
dimensions  of  product  or  technology  maturity.  One  writer  speaks  of  16 
different  dimensions  of  maturity.  The  TRL  measures  only  one  of  the  16. 
(Hobson,  2006,  p.  2) 

In  an  attempt  to  provide  a  more  complete  measure  of  technology  maturity  within 
the  Defense  Acquisition  Management  System,  the  GAO  (2006)  recommended  that  DoD 
expand  its  use  of  Technology  Transition  Agreements  (TTAs)  and  metrics  to  cover  aspects 
of  technology  maturity  not  explicitly  attended  to  by  the  TRL  (Appendix  C  summarizes 
the  management  of  technology  maturity  in  the  Defense  Acquisition  Management 
System).  Within  this  recommendation,  the  GAO  specifically  emphasized  the  utilization 
of  other  maturity  metrics,  such  as  the  Manufacturing  Readiness  Level  (MRL)  to  support 
better  management  of  technology  risk  and  to  avoid  the  pitfalls  of  proceeding  with 
immature  technologies  (GAO,  2006;  Appendix  D  details  the  impact  of  immature 
technologies).  Although  they  were  only  recently  introduced,  MRLs  have  already  gained 
wide  acceptance  throughout  government  and  industry  (AFRL/RXM,  2009). 

In  recent  years,  many  readiness  levels  like  the  MRL  have  surfaced.  Other 
examples  of  readiness  levels  developed  in  varying  degrees  include  the  Design  Readiness 
Level,  Integration  Readiness  Level,  and  System  Readiness  Level.  Ultimately,  the  efforts 
towards  creating  alternative  readiness  levels  have  been  to  replace,  amplify,  or 
compliment  existing  TRL  definitions  to  better  measure  and  manage  specific 
programmatic  areas  of  concern  (Sauser,  Ramirez-Marquez,  Magnaye,  &  Tan,  2008). 

C.  A  HUMAN-CENTERED  APPROACH  IS  NEEDED 

While  there  continues  to  be  an  increasing  number  of  TRL  variants  being  created, 
there  has  not,  however,  been  any  specific  effort  towards  capturing  the  socio-technical 
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attributes  of  technology  and  its  association  with  technology  readiness.  Without  a  human- 
centered  focus,  the  Human  Factors  Integration  (HFI)  Defence  Technology  Centre  (DTC) 
within  the  Ministry  of  Defence  (MoD)  states  that  programs  are  ill-prepared  to  identify, 
track,  and  resolve  human-related  issues  that  would  ensure  the  balanced  development  of 
both  human  and  technological  capabilities  (MoD  HFI  DTC,  2008). 

Historically,  the  lack  of  appropriate  Human  Systems  Integration  (HSI;  the  U.S., 
Canadian,  and  Australian  equivalent  to  HFI)  has  been  a  significant  stumbling  block  for 
many  programs,  even  those  with  supposedly  mature  technologies.  Whether  it  is 
redesigns,  substandard  system  performance,  or  dangerous  system  failures  endangering 
life  and  equipment,  poor  HSI  involvement  in  technology  maturation  can  be  catastrophic 
(Technology  Development  (TD)  1-12  Implementation  Team,  2008).  Two  unfortunate,  yet 
classic  examples  of  this  can  be  seen  in  Table  4: 

Table  4.  Three  Mile  Island  Incident/Loss  of  the  Mars  Climate  Orbiter  (After 

Technology  Development  (TD)  1-12  Implementation  Team,  2008) 

Three  Mile  Island  Incident _ 

On  March  28,  1979,  operators  at  Three  Mile  Island,  a  nuclear  power  plant  in 

Pennsylvania,  made  a  series  of  mistakes  that  led  to  a  near  meltdown  of  the  plant’s  reactor 

core.  This  accident  was  caused  by  a  cascade  of  equipment  failures  and  operator  errors. 

The  result  of  the  accident  was  a  release  of  approximately  1200  millirem/hour  of  radiation 

into  the  environment,  forcing  the  evacuation  of  several  thousand  residents  of  the 

surrounding  area.  Fortunately,  there  were  no  deaths  as  a  direct  result  of  the  incident.  The 

near  meltdown  of  the  reactor  occurred  when  an  emergency  relief  valve  at  the  top  of  the 

pressurizer  failed  to  close,  resulting  in  a  loss  of  the  pressurizer  steam  bubble  and  a  loss  of 

reactor  control  system  pressure  and  quantity.  The  indicator  lights  on  which  the  operators 

were  relying  to  determine  the  position  of  the  relief  valve  led  them  to  believe  that  the 

valve  was  closed.  However,  the  indicator  light  was  not  displaying  the  actual  system 

state — rather;  it  was  showing  the  presence  of  a  signal  commanding  the  valve  to  close.  In 

other  words,  the  operators  believed  the  relief  valve  was  closed  when  in  reality  the  valve 

was  open,  though  it  had  been  commanded  to  close.  This  led  the  system  operators  to 

conclude  falsely  that  a  leak  had  occurred,  and  they  began  to  act  accordingly. 
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The  operators  continued  to  make  errors  that  increased  the  volatility  of  the  system, 
such  as  confusing  reactor  B  with  reactor  A  (a  problem  directly  attributable  to  the  control 
panel  layout).  Not  until  two  hours  into  the  accident,  when  an  operator  who  had  recently 
arrived  finally  realized  that  the  relief  valve  was  at  fault  and  perfonned  the  proper  actions 
to  correct  the  problem.  In  the  end,  the  investigation  by  the  Nuclear  Regulatory 
Commission  into  the  human  factors  (HF)  aspects  of  the  accident  determined  that  the 
human  errors  which  occurred  during  the  incident  were  not  due  to  operator  deficiencies 
but  rather  to  inadequacies  in  equipment  design,  information  presentation,  emergency 
procedures,  and  training. 

Loss  of  the  Mars  Climate  Orbit er  (MCO) _ 

The  Mars  Climate  Orbiter  (MCO)  Mission  objective  was  to  orbit  Mars  as  the  first 

ever  interplanetary  weather  satellite  and  provide  a  communications  relay  for  the  Mars 
Polar  Lander  (MPL).  The  MCO  was  launched  on  December  11,  1998,  and  was  lost 
sometime  following  the  spacecraft's  Mars  Orbit  Insertion  (MOI)  maneuver.  The 
spacecraft's  carrier  signal  was  last  seen  at  09:04:52  UTC  on  Thursday,  September  23, 
1999. 

During  the  9  month  journey  from  Earth  to  Mars,  propulsion  maneuvers  were 
periodically  performed  to  remove  angular  momentum  buildup  in  the  on-board  reaction 
wheels  (flywheels).  These  Angular  Momentum  Desaturation  (AMD)  events  occurred  10- 
14  times  more  often  than  was  expected  by  the  operations  navigation  team.  This  was 
because  the  MCO  solar  array  was  asymmetrical  relative  to  the  spacecraft  body  as 
compared  to  Mars  Global  Surveyor  (MGS)  which  had  symmetrical  solar  arrays.  This 
asymmetric  effect  significantly  increased  the  Sun-induced  (solar  pressure-induced) 
momentum  build-up  on  the  spacecraft.  Additionally,  the  angular  momentum  (impulse) 
data  was  in  English,  rather  than  metric,  units.  This  measurement  confusion  resulted  in 
small  errors  being  introduced  in  the  trajectory  estimate  over  the  course  of  the  9  month 
journey.  At  the  time  of  Mars  insertion,  the  spacecraft  trajectory  was  approximately  170 
kilometers  lower  than  planned.  As  a  result,  MCO  either  was  destroyed  in  the  atmosphere 
or  re-entered  heliocentric  space  after  leaving  Mars'  atmosphere.  The  Board  recognized 
that  mistakes  occur  on  spacecraft  projects.  However,  sufficient  processes  are  usually  in 
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place  to  catch  these  errors  before  they  become  critical  to  mission  success.  Unfortunately 
for  MCO,  the  root  cause  was  not  caught  by  the  processes  in-place  in  the  MCO  project. 

The  MCO  Mishap  Investigation  board  (MIB)  determined  that  the  root  cause  of  the 
loss  of  the  MCO  spacecraft  was  the  failure  to  use  metric  units  in  the  coding  of  a  ground 
software  fde,  "Small  Forces",  used  in  trajectory  models.  Specifically,  thruster 
performance  data  in  English  units  instead  of  metric  units  was  used  in  the  software 
application  code  titled  SMFORCES  (small  forces).  The  AMD  file  contained  the  output 
data  from  the  SM  FORCES  software.  The  data  in  the  AMD  file  was  required  to  be  in 
metric  units  per  existing  software  interface  documentation,  and  the  trajectory  modelers 
assumed  the  data  was  provided  in  metric  units  per  the  requirements.  Inadequate  employee 
and  programmer  training  were  cited  as  the  reasons  for  the  omission  of  the  English-to- 
metric  conversion  factor  in  the  software  program  used  to  generate  the  AMD  files. 


The  two  examples  described  above  highlight  how  human  errors  can  result  in 
disastrous  events,  particularly  when  they  are  synergistically  compounded.  Therefore,  to 
reduce  errors,  maximize  performance,  and  enhance  safety,  proper  human-centered  design 
must  be  accomplished.  This  effort  must  attend  to  all  areas  that  impact  the  human  in  the 
system,  including  training,  source  selection,  human-computer  interaction,  ergonomics, 
safety,  survivability,  habitability,  quality  of  life,  and  human  performance  in  extreme 
environments  (e.g.,  space,  underwater,  and  Polar  expeditions;  O’Connor  &  Cohn,  2010). 

The  Air  Force  711th  Human  Performance  Wing  states  clearly  that  if  humans,  as 
part  of  an  integrated  human-technology  system,  are  not  considered  in  the  design  and 
implementation,  the  system  may  not  achieve  the  desired  operational  capability  (711 
HPW/HPO,  2009).  Expanding  on  this  belief,  the  UK  Vice  Admiral  Sir  Jeremy  Blackham 
stated: 


Capability  is  not  just  a  function  of  equipment  performance,  but  depends  on 
a  combination  of  interacting  elements.  Some  of  the  most  difficult  issues  to 
address  lie  in  the  human  factors  area.  The  types  of  systems  we  are 
specifying  and  procuring  now  will  shape  the  roles,  responsibilities  and 
career  paths  of  future  servicemen  and  women.  They  will  also  have  to  be 
operated  in  very  demanding  circumstances  of  fatigue,  hunger,  stress  and 
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even  fear,  by  the  sort  of  men  and  women  we  recruit.  They  will  therefore 
determine  not  just  the  working  environment  of  our  people,  but  ultimately 
their  utility  in  these  harsh  conditions  will  determine  our  operational 
success  and  our  ability  to  retain  the  right  people.  (MoD,  2007,  p.  4) 

Marine  Corps  General  James  Mattis,  head  of  Joint  Forces  Command  made  a 
similar  point  while  addressing  a  joint  war-fighting  conference.  Calling  on  industry  to 
focus  on  human-centered  design,  General  Mattis  was  quoted  saying,  "...  if  what  you’re 
doing  is  going  to  enable  the  human  interface,  then  you  ’re  on  the  right  track...  if  not,  you 
don ’t  want  things  that  take  geniuses  on  the  battlefield  to  operate.  We  need  to  create 
systems  and  organizations  and  equipment  that  don ’t  need  a  master ’s  degree  in  math  ” 
(Cavas,  2010,  para.  8). 

With  today’s  increasingly  interconnected,  diverse,  and  distributed  work 
environments,  the  focus  of  human-centered  design  and  technology  development  is  more 
important  now  than  ever.  Current  DoD  acquisition  policy  requires  optimizing  total 
system  performance  and  minimizing  the  cost  of  ownership  through  a  “total  system 
approach”  to  acquisition  management  (DoDD  5000.01,  2003).  As  seen  in  the  Three  Mile 
Island  incident,  as  well  as  the  loss  of  the  Mars  Climate  Orbiter,  the  total  system  includes 
not  only  the  prime  mission  equipment,  but  also  the  people  who  operate,  maintain,  and 
support  the  system;  the  training  and  training  devices;  and  the  operational  and  support 
infrastructure  (DAU,  2009).  The  DoDI  5000.02,  which  details  the  operation  of  the 
Defense  Acquisition  System  states: 

The  program  manager  (PM)  shall  have  a  plan  for  HSI  in  place  early  in  the 
acquisition  process  to  optimize  total  system  performance,  minimize  total 
ownership  costs,  and  ensure  that  the  system  is  built  to  accommodate  the 
characteristics  of  the  user  population  that  will  operate,  maintain,  and 
support  the  system.  (DoDI  5000.02,  2008,  p.  60) 

In  addition,  Chapter  6  of  the  Defense  Acquisition  Guidebook  (DAG)  states: 

The  HSI  domains  (manpower,  personnel,  training,  environment,  safety  and 
occupational  health,  human  factors  engineering,  survivability,  and 
habitability)  can  and  should  be  used  to  help  determine  and  work  the 
science  and  technology  gaps  to  address  all  aspects  of  the  system 
(hardware,  software,  and  human).  The  program  manager  should  integrate 
system  requirements  for  the  HSI  domains  with  each  other,  and  also  with 
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the  total  system.  As  work  is  done  to  satisfy  these  requirements,  It  is  vitally 
important  that  each  HSI  domain  anticipate  and  respond  to  changes  made 
by  other  domains  or  which  may  be  made  within  other  processes  or 
imposed  by  other  program  constraints.  (DAU,  2009,  section  6.2,  para.  1). 

Even  with  such  guidance,  incorporating  HSI  into  technology  and  system  life 
cycles  early  continues  to  be  a  challenge.  The  policies  that  are  currently  set  forth 
unfortunately  address  HSI  as  an  act  rather  than  a  process.  DoDI  5000.02,  for  example, 
thoroughly  defines  and  details  the  five  phases  of  the  Defense  Acquisition  Life  Cycle  (see 
Figure  4  in  Appendix  C);  however,  it  does  not  fully  integrate  or  define  HSI’s  role  within 
each  of  those  phases.  In  addition,  there  is  no  solid  guidance  on  the  preferred  sequencing 
of  needed  HSI  activities,  nor  are  there  any  HSI-specific  task  requirements  to  support 
adequate  accountability  and  implementation.  Without  an  appropriate  HSI  process  in 
place,  the  consistent  management  and  mitigation  of  human-related  risk  inherent  with  all 
developing  technologies  and  systems  will  continue  to  be  a  challenge. 

D.  EMERGENCE  OF  THE  HUMAN  VIEW 

Recognizing  the  abovementioned  need  for  HSI  involvement  and  a  sustainable 
human  approach  to  system  acquisition,  there  has  been  an  international  emergence  of  what 
is  being  tenned,  the  Human  View.  The  Human  View  is  a  supplementary  view  to  existing 
Architectural  Frameworks,  such  as  the  DoD  Architectural  Framework  (DoDAF)  that 
explicitly  models  the  human  elements  being  shaped  in  the  process  of  capability  design. 
By  doing  so,  HSI  is  considered  early  and  related  closely  to  the  design  and 
implementation  of  technology  (MoD  HFI  DTC,  2008).  The  ultimate  purpose  of  the 
Human  View  is  to  enable  effective  HSI  processes  within  the  design  of  complex,  large- 
scale,  socio-technical  systems. 

Architecture,  as  defined  by  DoD,  is  the  structure  of  components,  their 

relationships,  and  the  principles  and  guidelines  governing  their  design  and  evolution  over 

time  (DoDAF  VI. 5,  2007).  Architecture  frameworks,  such  as  DoDAF  are  used  within 

engineering  and  acquisition  communities  to  produce  common  approaches  to 

development,  presentation,  and  integration  of  current  and  future  systems.  The  DoDAF 

provides  the  guidance  and  rules  for  developing,  representing,  and  understanding 
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architectures  based  on  a  common  denominator  across  DoD,  Joint,  and  multinational 
boundaries.  It  provides  insight  for  external  stakeholders  into  how  the  DoD  develops 
architectures.  The  DoDAF  is  intended  to  ensure  that  architecture  descriptions  can  be 
compared  and  related  across  programs  and  mission  areas,  while  establishing  the 
foundation  for  analyses  that  supports  decision-making  processes  throughout  the  DoD 
(DoDAF  VI. 5,  2007). 


Within  DoD,  architectures  are  created  for  a  number  of  reasons.  From  a  practical 
perspective,  the  management  of  large  organizations  employing  sophisticated  systems, 
technologies,  and  services  in  pursuit  of  complex,  joint  missions  demands  a  structured, 
repeatable  method  for  evaluating  investments  and  investment  alternatives.  Architectures 
help  to  implement  organizational  change  effectively,  create  new  systems,  deploy  new 
technologies,  and  offer  services  which  add  value  to  decisions  and  management  practices 
(DoDAF  V2.0,  2009).  Newer  architecture  framework  versions,  such  as  DoDAF  V2.0 
address  Net-centric,  System  of  Systems,  and  System/Services  concepts  while 
emphasizing  data-centric  processes  to  support  DoD  managers  (as  process  owners  and/or 
decision-makers). 
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Figure  1.  Architecture  Viewpoints  in  DoDAF  V2.0  (From  DoDAF  V2.0  Vol.  1,  2009) 
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Figure  1  provides  a  graphical  representation  of  the  different  perspectives  or 
viewpoints  that  logically  combine  to  describe  system  architectures  in  DoDAF  V2.0.  It  is 
important  to  note  that  a  view  is  only  a  presentation  of  a  portion  of  the  total  architectural 
data,  in  the  sense  that  a  photograph  provides  only  one  view  of  the  object  within  the 
picture,  not  the  entire  representation  of  that  object.  The  viewpoints  are  organized  into 
eight  basic  sets  with  each  set  depicting  certain  architecture  attributes  as  described  in 
Table  5: 

Table  5.  DoDAF  V2.0  Viewpoint  Attributes  (After  DoDAF  V2.0  Vol.  1,  2009) 


Viewpoint 

Architecture  Attributes 

All  Viewpoint 

Provides  overarching  descriptions  of  the 
entire  architecture  and  defines  the  scope  and 
context  of  the  architecture 

Capability  Viewpoint 

Captures  the  goals  associated  with  the  overall 
vision  for  executing  a  specified  course  of 
action,  or  the  ability  to  achieve  a  desired 
effect  under  specific  standards  and  conditions 
through  combinations  of  means  and  ways  to 
perform  a  set  of  tasks 

Data  and  Information  Viewpoint 

Captures  business  information  requirements 
and  structural  business  process  rules,  and  it 
describes  the  information  that  is  associated 
with  information  exchanges 

Operational  Viewpoint 

Captures  the  organizations,  tasks,  or 
activities  performed,  and  information  that 
must  be  exchanged  between  them  to 
accomplish  DoD  missions 

Project  Viewpoint 

Captures  how  programs  are  grouped  in 
organizational  terms  as  a  coherent  portfolio 
of  acquisition  programs 

Services  Viewpoint 

Captures  system,  service,  and 

interconnection  functionality  providing  for, 
or  supporting,  operational  activities 

Standards  Viewpoint 

The  minimal  set  of  rules  governing  the 
arrangement,  interaction,  and 

interdependence  of  system  parts  or  elements 

Systems  Viewpoint 

Captures  the  information  on  supporting 
automated  systems,  interconnectivity,  and 
other  systems  functionality  in  support  of 
operating  activities 
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The  above  viewpoints  do  not,  however,  adequately  portray  the  human  as  a  unique 
part  of  the  system,  nor  do  they  capture  the  human-centered  design  aspects  needed  to 
ensure  the  effectiveness  of  human  operated  systems,  such  as  users’  requirements,  or 
capabilities  and  limitations  (Handley  &  Smillie,  2009).  It  is  due  to  this  deficiency  that  the 
Human  View  was  created. 

The  North  Atlantic  Treaty  Organization  (NATO)  Human  View  Handbook  states 
that  the  Human  View  forms  a  basis  for  decisions  by  stakeholders  by  providing  a 
structured  linkage  from  the  engineering  community  to  the  manpower,  personnel,  training, 
and  human  factors  engineering  communities.  It  provides  a  way  to  integrate  HSI  into  the 
mainstream  acquisition  and  systems  engineering  process  by  promoting  early  and  often 
consideration  of  human  roles.  Additionally,  it  provides  early  coordination  of  task  analysis 
efforts  by  both  system  engineering  and  HSI  teams.  Table  6  provides  an  overview  of 
NATO’s  current  set  of  Human  View  products  (greater  detail  of  all  Human  View  products 
and  their  potential  elements  can  be  found  in  Appendix  E). 

Table  6.  NATO  Human  View  Products  Overview  (After  Handley  &  Smillie,  2009) 


Human  View 

Description 

HV-A:  Concept 

A  conceptual,  high-level  representation  of  the 
human  component  of  the  enterprise  architecture 
framework. 

HV-B:  Constraints 

Sets  of  characteristics  that  are  used  to  adjust 
the  expected  roles  and  tasks  based  on  the 
capabilities  and  limitations  of  the  human  in  the 
system. 

HV-C:  Tasks 

Descriptions  of  the  human-specific  activities  in 
the  system. 

HV-D:  Roles 

Descriptions  of  the  roles  that  have  been  defined 
for  the  humans  interacting  with  the  system. 

HV-E:  Human  Network 

The  human  to  human  communication  patterns 
that  occur  as  a  result  of  ad  hoc  or  deliberate 
team  formation,  especially  teams  distributed 
across  space  and  time. 

HV-F:  Training 

A  detailed  accounting  of  how  training 
requirements,  strategy,  and  implementation 
will  impact  the  human. 
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Figure  2.  NATO  Fluman  View  Product  Relationships  (After  Handley  &  Smillie,  2009) 


The  NATO  Human  View  and  its  set  of  products  facilitate  design  decisions  by 
identifying  relevant  elements  or  specific  sets  of  data.  Relationships  can  be  defined 
between  the  products  that  express  dependencies  between  the  data.  Figure  2  depicts  some 
of  the  relationships  that  have  been  established  between  the  Human  View  products 
(Handley  &  Smillie,  2009). 

The  approach  NATO  has  taken  with  their  Human  View  framework  has 
emphasized  the  explicit  need  of  merging  seamlessly  and  efficiently  with  good  systems 
engineering  practice.  It  establishes  a  logical  and  systematic  framework  for  specifying  the 
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professional  data  collection  and  analysis  actions  needed  for  HSI  implementation  and  it 
makes  explicit  human,  crew,  and  team  socio-behavioral  processes  as  integral  to  total 
system  performance. 

An  example  of  the  Human  View  products  being  used  to  capture  human  elements 
can  be  seen  in  a  case  study  based  on  a  subset  of  the  U.S.  Army’s  Future  Combat  System 
(FCS).  The  FCS  program  was  a  modernization  initiative  designed  to  link  soldiers  to  a 
wide  range  of  weapons,  sensors,  and  information  systems  by  means  of  a  mobile  ad  hoc 
network  architecture  that  enables  joint  interoperability,  shared  situational  awareness,  and 
the  ability  to  execute  highly  synchronized  mission  operations.  It  consists  of  the  network, 
the  soldier,  and  14  systems,  including  eight  manned  ground  vehicles,  two  classes  of 
unmanned  aerial  systems,  two  classes  of  unmanned  ground  systems,  unattended  ground 
sensors,  and  the  Non-Line  of  Sight  Launch  System  (Handley  &  Smillie,  2009).  Table  7 
indicates  the  type  and  scope  of  infonnation  captured  in  each  of  the  Human  View  products 
for  the  FCS  example. 
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Table  7.  Content  of  Human  View  Products  for  FCS  Example 

(From  Handley  &  Smillie,  2009) 


Human  View  Products 

FCS  Case  Study  Products  Description 

HV-A  Concept 

A  graphic  that  shows  a  soldier  connected  through  a  network  to  14 
systems.  This  represents  the  FCS  “One  Soldier,  One  Network,  14 
Systems”  concept. 

HV-B  Personnel  Constraints: 
Manpower  Projections  (HV-B1) 

A  matrix  that  show's  the  distribution  of  the  50  personnel  in  an  Infantry 
Platoon  by  both  the  12  identified  roles  and  across  the  five  platoon 
vehicles. 

HV-B  Human  Factor 

Constraints:  Health  Hazards 
(HV-B5) 

A  table  that  shows  the  operating  limits  of  infantry  personnel  to  noise 
and  heat,  as  well  as  rest  requirements. 

HV-C  Tasks 

A  table  that  decomposes  the  platoon  level  tasks  of  Tactical  Road 
March  and  Reaction  to  Ambush  to  15  squad  level  tasks. 

HV-D  Roles 

A  table  listing  the  12  roles  defined  for  the  Infantry  Platoon  and  their 
required  Military  Occupational  Specialty  (MOS).  An  additional  table 
indicates  the  assignment  of  the  roles  to  the  previously  defined  tasks.  A 
third  table  indicates  the  controllers  and  system  interfaces  available  to 
each  role. 

HV-F.  Human  Network 

A  matrix  that  indicates  the  different  team  compositions  within  the 
platoon,  both  by  vehicle  and  by  functionality;  the  type  of  interactions 
i.e.,  collaboration,  coordination,  and  supervision,  are  also  indicated. 

For  example,  w  ithin  three  of  the  vehicles  are  two  rifle  squads  lead  by  a 
team  leader,  with  supervision  from  a  squad  leader.  Across  the  vehicles 
are  the  drivers  who  coordinate  maneuvers. 

HV-F  Training 

A  tabic  that  indicates  the  current  training  level  for  each  MOS 
type  indicated  in  the  role  table,  and  the  additional  training 
required  to  attain  the  knowledge,  skill,  and  abilities  to  operate 
the  FCS  systems. 

HV-G  Metrics 

A  listing  of  the  platoon  level  performance  metrics  for  conducting 
Counter  Ambush  Operations,  as  well  as  individual  squad  level 
performance  standards  based  on  FCS  specifications. 

The  subset  of  the  FCS  that  was  analyzed  included  the  capabilities  and  limitations 
of  an  infantry  platoon  and  the  resources  available  in  the  Infantry  Carrier  Vehicle  (one  of 
the  14  FCS  systems).  The  case  study  focused  on  the  tasks  involved  during  a  tactical  road 
march  and  the  platoon’s  reaction  to  an  enemy  ambush.  As  illustrated  above,  the  Human 
View  products  were  able  to  capture  the  different  roles  the  platoon  members  assumed 
during  these  operations  (i.e.,  platoon  leader,  driver,  etc.),  the  interactions  between  the 
platoon  functional  teams,  and  the  FCS  technology  interfaces.  Ultimately,  a  fully 
populated  and  developed  Human  View  would  reflect  the  product  of  iterative  analyses 


28 


describing  details  of  human  sensory,  perceptual,  emotional,  social,  cognitive,  ergonomic, 
biomechanical,  motor,  communicative,  and  decisional  processes  consistent  with  man- 
equipment  interactions. 

E.  SUMMARY 

In  order  to  effectively  translate  capability  needs  and  technology  opportunities  into 
stable,  affordable,  and  well-managed  acquisition  programs,  proper  measurement  and 
management  of  technology  maturity  must  occur.  The  TRL  has  proven  to  be  the  tool  of 
choice  for  describing  the  maturity  of  developing  technologies.  Yet,  due  to  fundamental 
limitations,  the  TRL  has  been  incapable  of  capturing  the  human-related  attributes  of 
technology  and  its  association  with  technology  readiness. 

The  need  for  a  better  fusion  of  human-centric  elements  is  not,  however,  unique  to 
only  technology  maturity  management.  This  issue  continues  to  surface  in  all  phases  of  the 
acquisition  life  cycle  and  it  is  in  that  very  reason  the  international  community  created  the 
Human  View.  The  Human  View,  which  is  a  complimentary  view  to  existing  architectural 
frameworks,  establishes  a  logical  and  systematic  structure  for  specifying  the  professional 
data  collection  and  analysis  actions  needed  for  HSI  implementation. 

The  next  chapter  introduces  the  first  iteration  of  the  proposed  Human  Readiness 
Level  and  its  attempt  towards  applying  the  technical  details  of  the  Human  View  and  other 
existing  sources  to  structure  HSI  initiatives  specifically  within  technology  maturity 
assessments.  The  steps  taken  in  developing  the  HRL  concept  will  first  be  discussed, 
followed  by  the  initial  validation  studies  in  subsequent  chapters. 
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III.  HUMAN  READINESS  LEVEL  DEVELOPMENT 


A.  OVERVIEW 

In  Defense  Acquisition  risk  management  structures,  developing  technologies  are 
monitored  and  managed  for  associated  risk  by  way  of  Technology  Readiness 
Assessments  (TRAs)  and  their  assignment  of  TRLs  (see  Appendix  C  for  more  details 
regarding  the  TRA  process).  The  lower  the  level  of  technology  readiness,  the  more 
ground  must  be  covered  to  bring  the  technology  to  the  point  at  which  it  can  meet  the 
intended  product’s  cost,  schedule,  and  performance  requirements  with  little  risk.  As 
mentioned  in  Chapter  II,  the  TRL  does  not  adequately  capture  socio-technical  elements 
of  developing  technology  and  without  such  attention;  risk  related  to  the  human  will 
continue  to  exist.  This  chapter  describes  the  development  of  the  first  iteration  of  the  HRL. 
The  method  used  in  creating  the  HRL,  as  well  as  the  resulting  framework  is  discussed 
below. 

B.  CREATING  THE  HRL  CONCEPT 

To  establish  the  critical  groundwork  for  developing  the  concept  of  the  HRLs, 
multiple  teleconferences  and  two  separate  five-day  workshops  took  place  at  the  711th 
Human  Performance  Wing/Human  Performance  Integration  Directorate  of  the  Air  Force 
Research  Laboratory.  The  development  team  consisted  of  the  thesis  author  and  Dr. 
Hector  Acosta,  HSI  professional  and  co-advisor  of  this  thesis. 

Step  1:  Literature  review 

The  first  step  was  to  conduct  an  extensive  literature  review  and  document 
analysis.  The  literature  reviewed  is  discussed  in  the  previous  chapter.  From  this  review  it 
was  decided  that  the  NATO  Human  View  was  the  most  appropriate  framework  on  which 
to  base  the  HRL.  The  reason  for  this  was  that  it:  a)  provided  a  structured  linkage  from  the 
engineering  community  to  the  technical  domains  of  HSI;  b)  it  established  a  logical  and 
systematic  framework  for  specifying  the  professional  data  collection  and  analysis  actions 


31 


needed  for  effective  HSI  implementation;  and  c)  it  provided  early  coordination  of  task 
analysis  efforts  by  both  system  engineering  and  HSI  teams. 

Step  2:  Establishing  the  underlying  principles 

For  the  HRL  framework  to  be  accepted  and  useful  to  the  HSI  and  acquisition 
communities,  four  underlying  principles  were  used  to  guide  the  development  of  the  HRL 
framework. 

1 .  Any  risk  management  framework  that  is  developed  for  HSI 
implementation  should  be  consistent  with  existing  risk  management  processes  and,  to  the 
extent  possible,  complement  existing  program  risk  management  metrics. 

2.  HSI  should  define  a  process  that  supports  the  development  and  fielding  of 
systems  that  minimize  life  cycle  costs  and  result  in  human-centered  designs  that 
complement  total  system  performance.  This  needs  to  be  done  in  a  way  that  is  consistent 
with  the  realities  of  system  engineering  risk  management  and  the  need  for  rational, 
informed  developmental  tradeoffs. 

3.  The  model  should  facilitate  vigorous  HSI  implementation  early  in  the 
acquisition  timeline  to  decrease  risk  of  resulting  life  cycle  cost  penalties. 

4.  In  the  context  of  a  competitive  cost,  schedule,  and  performance-driven 
acquisition  enterprise,  any  incremental  expense  to  implement  HSI  must  be  identified  in 
order  to  be  funded.  There  currently  is  no  process  in  DoD  that  identifies  the  incremental 
expenses  involved  with  implementing  HSI. 

Step  3:  Development  of  HRL  level  definitions 

The  decision  was  made  to  develop  nine  HRL  levels  to  be  consistent  with  the 
existing  TRL  structure.  The  reason  for  this  decision  was  to  encourage  acceptance  and  use 
among  the  S&T  and  acquisition  communities,  particularly  in  developmental  risk 
management  structures.  Each  of  the  nine  HRLs  was  given  a  preliminary  definition.  The 
following  list  provides  the  HRL  definitions  and  a  brief  rationale  for  each: 
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1.  Activation  of  Human  Systems  Integration:  base-lining  and  commitment. 

At  this  level,  the  priority  is  to  establish  HSI  infrastructure  to  support  and  manage  initial 

planning  and  analysis  efforts. 

2.  Human  Systems  Integration  analysis  suite  in  support  of  component 

technology  development.  The  goal  of  this  second  level  is  to  ensure  that  substantial 

preliminary  assessments  and  documentation  has  been  accomplished. 

3.  Component  human  touch  point  (i.e.,  human-system  interaction,  to  include 
hardware  and  software)  resolution:  refining  requirements  thresholds.  The  primary 
objective  of  this  level  is  to  continue  HSI  analysis  efforts  across  all  domains,  while 
identifying  and  communicating  appropriate  technological  thresholds. 

4.  Component  human  touch  point  engineering  parameters  and  human 
performance  indicators.  The  fourth  level  continues  with  iterative  analyses  and  updates  to 
HSI  domain  considerations  bearing  on  the  developing  technology. 

5.  Limited  system  human  performance  parameters/demonstration.  In  this 
level,  HSI  activities  focus  on  supporting  the  increasing  fidelity  of  breadboard  technology. 

6.  Field  (relevant  environment)  validation  of  human  performance  prototypes. 
The  goal  of  this  sixth  level  is  to  make  certain  significant  HSI  evaluations  are  conducted 
in  relevant  environments. 

7.  Final  Developmental  Test  &  Evaluation/human  performance  parameters. 
The  objective  for  this  level  is  to  ensure  HSI  involvement  in  system  T&E. 

8.  Operational  Test  &  Evaluation /human  performance  parameters.  This 
level  focuses  on  accomplishing  specific  HSI  analyses  to  support  OT&E  events. 

9.  Sustainment:  Initiation  of  capability  gap  feedback  cycle.  The  goal  of  the 
ninth  and  final  level  is  to  begin  iterative  review  and  verification  of  fielded  systems  to 
support  ongoing  sustainment. 

Step  4:  Development  of  more  detailed  level  descriptions 

The  definitions  developed  in  the  previous  stage  established  the  framework  to 

allow  for  the  more  detailed  development  of  the  HRLs.  This  was  accomplished  by 
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elaborating  the  products  of  the  Human  View  and  explicitly  developing  their  linkage  to 
the  concept  of  an  HRL.  There  were  five  sub-steps  carried  out  at  this  stage  of  the 
development  process: 

1.  An  exhaustive  list  of  HSI  goals  and  considerations  was  developed. 

2.  The  HSI  goals  and  considerations  were  arranged  sequentially  based  upon  the 
location  in  the  acquisition  life  cycle. 

3.  The  unique  and/or  the  more  often  iterative  activities  required  to  collect  HSI 
data  were  identified. 

4.  Data  types  were  translated  into  language  across  the  HSI  domains,  targeting 
clearly  specified  requirements  and  acquisition  documents  as  they  evolve  (with  the 
understanding  that  all  data  converges  toward  specifications). 

5.  Data  collection  activities  were  linked  with  the  standardized  professional 
activities  and  process  titles  (e.g.,  Initial  HSI  Assessment,  Hazard  Analyses,  Cognitive 
Task  Analyses)  that  need  to  be  executed  by  HSI  practitioners  in  the  collection  of  valid, 
reliable,  and  accurate  HSI  data. 

Step  5:  Synthesis  of  the  HRLs  with  the  Human  View 

The  output  from  step  4  of  the  HRL  development  was  synthesized  with  the  Human 
View,  cross-referenced  with  the  HSI  Integrated  Framework  Version  1.3,  and  entered  into 
a  Microsoft  Excel  Workbook.  The  HRL  framework  is  summarized  in  Table  8,  with  the 
much  more  detailed  spreadsheet  included  in  Appendix  F. 

Table  8.  Human  Readiness  Level  Framework  Overview 


Human  Readiness  Level 

Description 

1 .  Activation  of  Human  Systems 

Integration:  base-lining  and  commitment. 

Lowest  level  of  socio-technical  readiness. 
HSI  infrastructure  is  setup  within  Systems 
Engineering  planning.  Total  system 
analysis  from  both  functional  relationship 
and  organizational  perspectives  occurs. 
Activity  examples  include  front-end 
analyses,  preliminary  functional  allocation, 
and  initial  HSI  assessment  and  plan. 
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2.  Human  Systems  Integration  analysis 
suite  in  support  of  component  technology 
development. 

Significant  HSI  input  to  acquisition 
development  and  documentation  occurs. 
Activity  examples  include  initial  human- 
machine  interface  assessment,  generation 
of  Target  Audience  Description,  and 
threat/hazard  assessment. 

3.  Component  human  touch  point 
resolution  (i.e.,  human-system  interaction, 
to  include  hardware  and  software): 
refining  requirements  thresholds. 

Multiple  needs  analyses  and  studies  are 
conducted  in  support  of  requirement 
definitions.  HSI  domain  assessments 
inform  ongoing  development  actions,  as 
well.  Activity  examples  include  human-in- 
the-loop  analyses,  sub-system  hazard 
analysis,  and  HSI  plan  update  and  revision. 

4.  Component  human  touch  point 
engineering  parameters  and  human 
performance  indicators. 

Iterative  evaluation  and  analysis  of  each 
HSI  domain  takes  place  and  provides 
critical  items  of  consideration  bearing  on 
system  design.  Activity  examples  include 
usability  testing,  development  of  human- 
centered  source  selection,  and  updating  the 
human-machine  interface  assessment. 

5.  Limited  system  human  performance 
parameters/ demonstration. 

Various  HSI  assessments  and  testing  are 
performed  to  support  the  significant 
increase  in  fidelity  of  breadboard 
technology.  This  includes  supporting  “high 
fidelity”  laboratory  integration  of 

components.  Activity  examples  include 
examining  safety  and  occupational  health 
design  features  and  cognitive  task  analyses. 

6.  Field  (relevant  environment)  validation 
of  human  performance  prototypes. 

Representative  model  or  prototype  system 
is  tested  in  a  relevant  environment. 
Evaluations  of  human  perfonnance 
embedded  in  demonstration  system  occur 
and  HSI  predictive  models  are  updated. 
Activity  examples  include  testing  human 
reliability  and  usability  of  prototype  in  a 
high-fidelity  laboratory  environment  or  in 
simulated  operational  environment. 
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7.  Final  Developmental  Test  & 
Evaluation/human  performance  parameters. 

Significant  HSI  participation  in  system  test 
events  occurs.  Iterative  evaluation  and 
analysis  of  each  HSI  domain  continues  as 
well.  Activity  examples  include  error  and 
fault  analysis  to  cover  human  error 
perfonnance,  equipment  operability,  safety 
procedures,  and  error  recovery 

mechanisms. 

8.  Operational  Test  &  Evaluation/human 
performance  parameters. 

Special  human-centric  analyses  are 
conducted  to  update  thresholds,  objectives, 
and  evolving  criteria  for  Operational  Test 
&  Evaluation.  Iterative  evaluation  and 
analysis  of  each  HSI  domain  continues  as 
well.  Activity  examples  include  system 
hazard  analysis  and  HSI  domain  tradeoff 
studies. 

9.  Sustainment:  Initiation  of  capability  gap 
feedback  cycle. 

Extensive  and  iterative  review  and 
verification  of  fielded  system  begins,  as 
well  as  post-product  improvement 
evaluations  for  the  next  incremental  builds. 
Activity  examples  include  post-fielding 
training  evaluation  analysis  and  sustaining 
a  hazard  analysis  for  the  fielded  system. 

The  HSI  Integrated  Framework  Version  1.3  was  developed  by  the  U.S.  Navy’s 
Space  and  Naval  Warfare  Systems  Command  to  provide  specific  guidance  on  how  to 
integrate  HSI  processes,  products,  and  tools  into  Defense  Acquisition  (Risser,  Belk, 
Smillie,  Gepp,  2009).  The  HSI  Integrated  Framework  Version  1.3  was  used  specifically 
to  expand  the  HRL  framework  with  any  relevant  detail  not  previously  included.  This 
allowed  the  final  decisions  to  be  made  regarding  the  HRL’s  functional  structure, 
processes,  and  intent. 

C.  LINKING  THE  HRL  FRAMEWORK  WITHIN  THE  ACQUISITION  LIFE 
CYCLE 

The  intent  was  for  each  HRL  to  provide  a  risk  management  metric  that  follows  a 
scale  from  1  ( HSI  base-lining  &  commitment)  to  9  ( capability  gap  feedback  cycle).  Risk, 
for  purposes  of  HRLs,  is  mitigated  by  professional  analytical  and  managerial  activities 
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that  link  critical  items  of  consideration  to  domain-specific  design  decisions,  including 
well-informed  tradeoffs.  A  program  determined  to  be  at  HRL-1  is  at  the  lowest  level  of 
HSI  readiness  (thus,  highest  level  of  HSI  risk),  where  the  initial  activation  of  HSI 
commitment  and  processes  occurs.  By  the  time  the  program  has  reached  HRL-9,  it  has 
progressed  through  significant  HSI  analyses,  requirement  threshold  refinements,  field 
validations  of  human  performance  prototypes,  and  extensive  developmental  and 
operational  Test  and  Evaluation  (T&E)  of  human  performance  parameters.  Ultimately,  a 
program’s  human-centered  maturity  is  achieved  through  the  performance  of  HSI 
activities  which  address  the  consideration  of  end-users  and  other  stakeholders  in  the 
specification,  development,  and  operation  of  a  system. 

Each  HRL  level  provides  a  description  of  the  type  of  actions  required  to  take  the 
next  step  towards  increasing  the  socio-technical  maturity  of  a  program  with  respect  to 
those  milestone-sensitive  phases  in  a  technology’s  life  cycle.  For  instance,  the  designated 
criterion  threshold  to  be  met  at  Milestone  A  in  the  Defense  Acquisition  Life  Cycle 
Management  System  is  HRL-2;  HRL-6  should  be  achieved  prior  to  Milestone  B;  and 
HRL-8  prior  to  Milestone  C.  Figure  3  illustrates  the  HRL  milestone  progression  below 
(the  milestones  are  also  discussed  in  more  detail  in  Appendix  C). 
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Figure  3.  HRL  Milestone  Progression  in  the  Defense  Acquisition  Life  Cyle 

(After  DoDI  5000.02,  2008) 
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By  structuring  appropriate  sequencing  of  HSI  activities  and  tracking  the  progress 
of  HSI  planning  and  execution,  the  HRL  distinguishes  itself  as  a  significant  risk 
management  tool  for  process  owners  and  decision  makers  in  Defense  Acquisition.  Unlike 
TRLs  where  technology  maturity  descriptors  serve  as  the  basis  for  level  assignment,  the 
HRL  functions  by  basing  its  HSI-related  risk  and  maturity  level  off  of  the  execution  and 
outcome  of  those  milestone-sensitive  management  and  analytical  activities  that  have  been 
designated  in  each  progressive  HRL.  Higher  HRL  levels  are  intended  to  be  consistent 
with  more  complete  specification,  including  the  effects  of  tradeoff  decisions.  This  also 
serves  to  differentiate  the  HRL  from  the  Human  View.  For  instance,  the  Human  View  has 
been  tailored  to  architectural  models  and  serves  as  a  repository  for  required  HSI  data.  It 
does  not,  however,  provide  the  data.  It  is  only  by  doing  the  unspecified  and  carefully 
crafted  data  collection  and  analyses  delineated  within  tools  such  as  the  HRL  that  HSI  data 
can  be  produced. 

D.  SUMMARY 

This  chapter  described  the  process  by  which  the  Human  Readiness  Level  was 
developed.  The  next  chapter  builds  upon  these  efforts  by  describing  a  subject  matter 
expert  (SME)  evaluation  of  the  HRL.  The  focus  of  this  initial  evaluation  will  be  on  the 
HRL  framework’s  overall  worth  and  usability. 
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IV.  HUMAN  READINESS  LEVEL  EVALUATION 


A.  BACKGROUND 

In  order  for  the  HRL  to  be  incorporated  within  DoD  as  a  legitimate  HSI  planning 
tool  and  complementary  (HSI-specific)  metric  in  program  risk  management  structures,  it 
is  necessary  for  it  to  first  be  evaluated  and  scrutinized  by  the  community  to  which  it  has 
been  developed  for.  This  chapter  describes  an  initial  evaluation  of  the  HRL  framework  by 
SMEs  familiar  with  HSI  and  the  DoD  acquisition  process. 

B.  METHODOLOGY 

1.  Instrument 

A  questionnaire  was  designed  to  gain  feedback  as  to  the  overall  worth  and 
potential  (i.e.,  accuracy,  ease  of  use,  completeness)  of  the  HRL  framework  described  in 
the  previous  chapter.  The  questionnaire  consisted  of  51  items  divided  into  five  separate 
groups.  The  first  four  grouping  of  items  were  designed  to  obtain  feedback  on  the  HRL’s 
recommended  milestone  progression  in  the  Defense  Acquisition  Life  Cycle.  For  instance, 
HRL-2  is  the  criterion  threshold  to  be  met  at  Milestone  A,  therefore  the  first  set  of  items 
pertained  to  HRLs  1  and  2.  The  last  group  of  items  concerned  the  proposed  categories  for 
future  HRL  framework  efforts.  Both  the  HRL  framework  and  the  evaluation  instrument 
can  be  viewed  in  Appendix  F  and  G  respectively. 

Data  were  gathered  from  participants’  responses  to  the  HRL  review  questionnaire 
based  on  their  ratings  of  agreement  from  the  five-point  Likert  scale  ranging  from 
“strongly  agree”  to  “strongly  disagree.”  The  corresponding  responses  to  the  statements 
were  converted  to  a  numerical  value  ranging  from  1  to  5  (l-“strongly  agree”  to 
5-“strongly  disagree”).  In  addition,  within  all  five  sections,  the  participant  had  the 
opportunity  to  make  open-ended  comments. 
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2.  Participants 


To  obtain  feedback  from  SMEs,  the  selection  criteria  for  inclusion  in  the  study 
was  that  the  participants  had  to  have  experience  and/or  graduate  level  training  in  both 
HSI  and  Defense  Acquisition.  Consideration  was  given  to  focusing  solely  on  the 
professionals  working  in  these  areas  within  the  Air  Force.  This  was  due  to  the  Air  Force 
HSI  community  being  accessible  to  the  researcher.  However,  because  of  the  limited 
expertise  in  this  combined  area,  it  was  decided  that  the  evaluation  should  be  expanded  to 
other  services.  Therefore,  the  questionnaire  was  distributed  to  42  individuals  within  DoD, 
as  well  as  one  individual  from  the  Canadian  military.  Because  the  SMEs  served  as 
participants  to  the  research  and  were  not  subjects  of  the  research,  the  Naval  Postgraduate 
School’s  Institutional  Review  Board  (IRB)  detennined  the  study  protocol  did  not  require 
IRB  involvement. 

C.  RESULTS 

Out  of  the  43  identified  SMEs,  a  total  of  15  HRL  evaluations  were  returned  (a 
35%  response  rate).  Of  those  who  responded,  one-third  represented  SMEs  from  the  Air 
Force  (N=5),  while  the  remaining  two-thirds  consisted  of  respondents  from  other 
branches  of  the  military  (Army=2,  Navy=4,  Marines=2,  Coast  Guard=l,  Canadian 
military=l).  As  the  HRL  framework  was  developed  by  two  Air  Force  individuals,  it  was 
important  to  examine  whether  there  were  differences  in  the  levels  of  satisfaction  with  the 
system  between  Air  Force,  and  non-Air  Force  respondents.  Table  9  summarizes  the  mean 
(and  standard  deviation)  of  the  levels  of  satisfaction  for  five  groups  of  items.  It  can  be 
seen  that  the  level  of  satisfaction  of  both  Air  Force  and  non-Air  Force  affiliated 
respondents  improved  with  each  higher  level  of  the  HRL  framework. 
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Table  9.  Rating  of  Agreement  Summary  for  the  SME  Evaluation  of  the  HRL 

Framework  (1-  “Strongly  Agree”  to  5  -  “Strongly  Disagree”) 


SME  Evaluation  Report 

Affiliation 

HRFs  1/2 

HRFs  3/4/5/6 

HRFs  7/8 

HRF-9 

Proposed 

Categories 

Air  Force 

Mean 

3.04 

2.76 

2.49 

2.49 

2.16 

Std.  Deviation 

0.66 

0.63 

0.54 

0.54 

0.21 

N 

5 

5 

5 

5 

5 

Other 

Mean 

2.33 

2.05 

1.97 

1.88 

1.89 

Std.  Deviation 

0.48 

0.39 

0.37 

0.38 

0.33 

N 

10 

9 

9 

9 

8 

Total 

Mean 

2.57 

2.30 

2.15 

2.10 

1.99 

Std.  Deviation 

0.63 

0.51 

0.48 

0.52 

0.31 

N 

15 

14 

14 

14 

13 

The  Mann  Whitney  U  test  (the  non-parametric  equivalent  of  the  between  subjects 
t-test)  was  used  to  assess  whether  there  was  a  significant  difference  between  Air  Force 
affiliated  participants,  and  those  from  other  services.  This  analysis  was  carried  out  by 
examining  the  mean  level  of  satisfaction  for  the  items  corresponding  to  each  of  the  five 
sections  of  statements. 

Although  the  non-Air  Force  affiliated  respondents  were  more  satisfied  in  each 
category,  the  difference  between  the  two  groups  was  only  significant  for  the  items 
associated  with  HRFs  3  to  6  (Z=  2.0,  p<.05). 
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The  Friedman’s  rank  sign  test  (the  nonparametric  equivalent  of  the  repeated 
measures  Analysis  of  Variance)  was  used  to  assess  whether  there  were  significant 
differences  in  the  satisfaction  of  participants  with  each  of  the  four  groups  of  HRLs.  The 
test  was  found  to  be  significant  (chi  square=  9.0,  df=  3,  p<.05).  The  respondents  were 
significantly  more  satisfied  with  the  higher  HRLs  than  the  lower  HRLs. 

A  summary  of  the  open-ended  comments  made  by  the  respondents  is  provided 
below  (see  Appendix  I  for  complete  list  of  all  of  the  comments  made). 

HRLs  1  and  2 

For  the  first  section  of  statements  regarding  HRLs  1  and  2,  many  participants 
stated  that  both  HRLs  seemed  reasonably  aligned  with  the  acquisition  timeline  and 
function.  However,  in  regards  to  whether  any  critical  infonnation  was  missing,  many 
commented  that  acquisition  programs  will  likely  require  activities  or  infonnation  not 
necessarily  found  in  this  first  version  of  the  framework.  Typical  comments  included: 

“These  timings  are  probably  program-dependent,  but  they  seem  reasonably 
aligned  with  the  DODI  5000.02  timeline...  ” 

“We  don't  know  what  we  don't  know.  We  may  find  that  there  are  critical  drivers 
that  are  not  under  our  purview  or  within  our  awareness  on  any  given  program...  ” 

HRLs  3,  4,  5,  and  6 

Many  comments  made  within  this  section  cited  a  need  for  more  detail  and  better 
defined  descriptions.  Typical  comments  made  were: 

“Stating  HRLs  3-5  all  should  occur  “as  soon  as  possible  after  MS  A  ”  does  not 
help  discriminate  between  the  three.  Timing  for  activity  completion  should  be 
more  defined.  ” 

“Descriptors  should  better  distinguish  maturity  levels,  particularly  HRLs  3  &  4.  ” 

“Reference  to  “component”  in  HRLs  3&4  can  mislead  that  system  level  was 
bypassed  or  circumvented.  ” 
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HRLs  7  and  8 


Of  the  various  comments  regarding  HRLs  7  and  8,  most  were  specific 
recommendations  for  improving  different  aspects  of  the  HRL  framework.  These 
comments  included: 

“Need  to  find  a  way  to  take  findings  from  DT&E  and  inform  /  support  OT&E  test 
readiness  evaluation  /  report.  Almost  all  HRL  testing  can  be  accomplished 
unobtrusively  in  DT&E  and  in  enough  detail  to  know  what  to  expect  in  OT&E. 
HRL  determination  in  OT&E  will  have  to  be  a  natural  outcome  from  the  actual 
tests — it  must  be  unobtrusive.  ” 

“ The  analysis  by  CDR  for  HRL-7  needs  to  be  pretty  comprehensive  as  the  design 
is  pretty  much  locked  in  after  this  point  and  changes  become  difficult  to 
influence/implement.  Appears  activities  for  HRLS  should  be  better  tailored 
toward  what  resulted  from  DT&E,  and  how  will  deficiencies  be  addressed  prior 
to  OT&E.  ’’ 

HRLS 

This  section  provided  a  limited  amount  of  comments,  and  the  comments  did  not 
appear  to  be  centered  around  any  particular  theme.  Example  comments  included: 

“ Recommend  this  HRL/description  be  aligned  with  Post  Implementation  Review.  ” 

“It  is  important  to  include  feedback  throughout  the  entire  acquisition  process,  not 
just  during  HRL  9.  ” 

Proposed  Categories 

Each  of  the  seven  proposed  categories  contained  comments  that  indicated  relative 
satisfaction  and  agreement.  Typical  comments  made  for  each  proposed  category  include 
the  following: 
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Acquisition  &  Sustainment  Toolkit  Linkages 

“ Since  Logistics  considerations  should,  like  manpower  and  training  issues,  be 
dealt  with  early,  it  is  not  unlikely  that  much  lower  HRL  levels  could  be  influenced 
by  the  ASTK.  ” 

Integration  Notes  and  Comments 

“Agreed,  additional  details  will  be  especially  useful  for  individuals  tasked  with 
HSI  that  are  not  very  familiar  with  it  or  with  the  acquisition  process.  ” 

Target  Documents 

“Agreed,  individuals  can  go  look  at  those  documents  for  additional  detail  on 
program.  ” 

“Recommend  considering  list  of  target  documen  ts  for  NEXT  phase  here  as  well  to 
highlight  upcoming  tasks  for  planning  purposes.  ” 

Action  OPR 

“There  should  be  a  program  office  OPR,  a  Sponsor/MAJCOM  OPR ,  and  an  HSI 
OPR.  ” 

“ Especially  when  individuals  move  on,  it  is  nice  to  have  a  POC  to  go  back  to  or 
identify  that  the  work  has  been  in  process.  ” 

References 

“Will  provide  information  to  individuals  performing  the  tasks  on  exactly  what 
they  need  to  do  and  where  to  go  to  find  information.  ” 

“Consider  hyper  linking  or  a  separate  reference  database/ dictionary.  ” 

Products  of  Activity 

“Strongly  agree...  for  building  off  of  effort  in  next  HRL  or  activity  ” 

Rough  Cost  Estimate 

“This  provides  PM  with  information  on  the  cost  of  doing  the  HSI  at  each 
level/activity  for  a  program.  ” 
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“This  will  allow  PM  to  pick  and  choose  the  most  important  activities  based  on 
budget  restrictions.  ” 


D.  DISCUSSION 

In  general,  larger  sample  sizes  permit  greater  confidence  in  the  results  and  can 
increase  the  probability  of  uncovering  a  wide  range  of  underlying  issues  being  studied. 
However,  as  stated  by  Krosnick  (1999):  “ recent  research  has  shown  that  surveys  with 
very  low  response  rates  can  be  more  accurate  than  surveys  with  much  higher  response 
rate ”  (p.  540).  The  representativeness  is  more  important  than  the  sample  size.  Given  that 
there  is  not  a  large  number  of  individuals  within  DoD  who  are  experienced  in  both 
acquisition  and  HSI,  it  was  not  possible  to  obtain  a  large  number  of  responses  from  the 
‘right’  people.  Valuable  insight  and  key  information  was  still  gleaned  from  a  large 
proportion  of  the  community  who  will  be  using  the  HRL  framework.  The  following 
paragraphs  discuss  the  findings  from  the  quantitative  and  qualitative  data. 

HRLs 

The  agreement  ratings  and  comments  regarding  the  four  separate  groups  of  HRLs 
indicated  that  the  respondents  were  generally  satisfied  with  the  framework.  However,  a 
number  of  suggested  changes  were  made.  For  the  first  group  of  HRLs  1  and  2,  the  typical 
concern  was  that  the  framework  would  likely  require  more  activities  or  information  to  be 
truly  complete.  This  was  not  unexpected,  particularly  because  of  the  sheer  amount  of 
actions  and  activities  needed  to  address  all  HSI  domains.  Although,  the  intent  is  to 
include  an  exhaustive  list  of  HSI  activities  within  the  HRL  framework,  the  list  is 
understandably  not  yet  complete  in  this  first  version  of  the  framework. 

For  the  remaining  three  groups  (HRLs  3/4/5/6,  HRLs  7/8,  and  HRL  9),  most 
comments  made  by  the  SMEs  cited  a  need  for  more  detail  and  better  defined  descriptions. 
Of  particular  importance  was  the  timing  of  HSI  activities  contained  within  each  separate 
HRL.  The  concern  was  that  the  schedule  for  activity  completion  was  not  always  clear. 
For  example,  currently  the  HRL  framework  states  that  HRLs  3  through  5  should  all  occur 
“as  soon  as  possible  after  MS  A,  prior  to  MS  B.”  This  statement  of  timing,  according  to 
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one  SME,  did  not  help  to  discriminate  between  the  three  HRLs.  Future  improvements 
will  need  to  focus  on  the  explicit  delineation  of  HRLs  and  their  descriptions  with  regards 
to  acquisition  timing  and  milestone  events. 

Other  findings  involved  significant  differences  in  the  satisfaction  of  participants 
with  each  of  the  four  groups  of  HRLs.  Specifically,  the  Friedman’s  rank  sign  test 
revealed  that  as  the  levels  within  the  HRL  framework  increased,  so  did  the  respondents’ 
level  of  satisfaction.  However,  other  than  the  ratings  of  agreement  that  were  received,  the 
open-ended  comments  did  not  reveal  any  distinctive  reasoning  for  the  SMEs’  preference 
of  the  higher  HRLs. 

Taken  at  face  value,  their  lower  satisfaction  of  the  earlier  HRLs  could  simply 
indicate  more  effort  and  research  should  be  placed  on  these  levels.  However,  this  would 
ignore  other  potential  explanations  worthy  of  consideration.  For  instance,  Krosnick 
(1999)  differentiates  between  two  different  strategies  for  responding  to  questionnaire 
items:  optimizing  and  satisficing.  Optimizing  requires  the  respondent  to  put  forth 
cognitive  effort  in  order  to  generate  the  optimum  answer.  Conversely,  when  using  a 
satisficing  strategy,  rather  than  expending  the  effort  to  generate  optimal  answers, 
respondents  may  compromise  their  standards  and  expend  less  energy.  Satisficing  is 
especially  common  when  the  respondents  are  required  to  answer  a  long  series  of 
questions  on  a  wide  range  of  topics  (as  with  the  HRL  evaluation).  Therefore,  it  is  very 
possible  that  although  the  respondents  were  initially  motivated  to  provide  high-quality; 
optimized  feedback  at  the  beginning  of  the  HRL  evaluation,  they  may  have  become 
increasingly  fatigued  as  the  questionnaire  progressed  and  shifted  to  a  satisficing  response 
strategy. 

Another  interesting  finding  was  that  non- Air  Force  affiliated  respondents  were 
more  satisfied  in  each  category  of  the  HRL  framework  (although  the  difference  was  only 
significant  for  the  items  in  HRLs  3  to  6).  Although  the  reason  for  this  finding  is  difficult 
to  ascertain  without  conducting  post  interviews,  one  reason  could  be  that  the  HRL 
concept  originated  within  the  Air  Force.  Being  more  familiar  with  the  idea  of  an  HRL, 
the  Air  Force  SMEs  likely  had  a  vested  interest  and  were  more  critical  in  their  responses. 
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Proposed  Categories 

The  final  section  of  the  HRL  evaluation  was  focused  on  the  eight  proposed 
categories  for  future  framework  development  efforts.  All  eight  categories  received  many 
“strongly  agree”  and  “agree”  ratings  from  the  SMEs.  Along  with  several  open-ended 
comments,  the  general  feeling  suggested  that  the  proposed  additions  to  the  HRL 
framework  is  very  relevant  and  will  be  useful  once  defined  and  fused  into  the  HRL 
framework  matrix.  Of  all  of  the  proposed  additions,  the  Rough  Cost  Estimate  category 
received  the  most  attention  (i.e.,  number  and  length  of  comments).  However,  this  was  not 
surprising  due  to  the  cost-driven  nature  of  most  acquisition  environments. 

E.  SUMMARY 

The  results  of  this  evaluation  suggested  relative  agreement  among  SMEs  that  this 
first  iteration  of  the  HRL  concept,  once  fully  developed,  will  serve  as  a  valuable  HSI 
planning  tool  and  complementary  (HSI-specific)  metric  in  program  risk  management 
structures.  In  order  to  improve  the  framework,  many  SMEs  recommended  expanding  the 
HRL’s  list  of  activities  to  account  for  a  more  diverse  variety  of  programs  that  exist  in 
acquisition.  Suggestions  like  these  will  be  documented  to  form  a  basis  of  advocacy  for 
future  HRL  development. 

This  effort  has  examined  the  overall  worth  of  the  HRL;  however,  it  is  only  the 
first  step  in  a  continuing  process  of  definition  and  validation.  More  professional  scrutiny 
is  needed.  Therefore,  the  next  chapter  focuses  on  the  validation  of  the  HRL  model  when 
being  practically  applied  in  a  current  acquisition  program. 
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V.  A  CASE  STUDY  OF  THE  USEFULNESS  OF  THE  HRL 
FRAMEWORK  AS  APPLIED  TO  THE  GROUND  CONTROL 
STATION  MODERNIZATION  PROGRAM 


A.  BACKGROUND 

In  Chapter  III,  the  initial  SME  evaluation  of  the  HRL  framework  was  described. 
However,  that  analysis  was  not  applied  to  a  particular  acquisition  program.  In  this 
chapter,  a  case  study  of  the  application  of  the  initial  HRL  framework  was  carried  out  as 
used  by  a  SME  with  reference  to  a  specific  acquisition  program.  Specifically,  the  HRL 
framework  was  evaluated  to  assess  whether  it  contributes  to  the  creation  and  sustainment 
of  an  acquisition  program’s  HSI  Plan  (HSIP).  The  program  chosen  for  this  case  study 
was  the  U.S.  Air  Force  Ground  Control  Station  Modernization  (GMOD)  program. 
Background  regarding  the  GMOD  and  the  HSIP  will  be  presented,  followed  by  the 
evaluation  of  the  potential  usefulness  of  the  HRL  in  this  particular  context. 

1.  GMOD 

The  GMOD  program  is  an  acquisition  currently  being  developed  in  the  Air  Force 
to  advance  the  capabilities  of  legacy  Ground  Control  Stations  (GCSs).  GCSs  are  the 
control  centers  that  provide  the  facilities  for  human  control  of  remotely  piloted  aircrafts 
(RPAs).  The  standard  GCS  consists  of  pilot  and  sensor  operator  workstations,  as  well  as 
required  support  equipment.  Currently,  the  GMOD  program  is  using  a  phased  capabilities 
development  approach  where  each  phase  retrofits  the  legacy  GCS  with  safety-critical, 
mission-critical,  and  reliability  and  maintainability  capabilities.  The  ultimate  goal  for  the 
GMOD  program  is  to  significantly  improve  human-machine  interfaces  and  ergonomics  to 
increase  RPA  mission  capability. 

2.  HSIP  for  GMOD 

The  key  to  a  successful  HSI  acquisition  strategy  for  GMOD  is  comprehensive 
integration  across  the  HSI  domains  and  also  across  other  core  acquisition  and  engineering 
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processes.  Ultimately,  this  integration  is  dependent  on  an  accurate  HSIP  (DAU,  2009). 
The  HSIP  is  the  management  tool  used  to  plan,  manage,  and  implement  HSI  in  an 
acquisition  program.  It  is  the  essential  plan  used  in  identifying  HSI  issues  and 
recommending  resolutions  for  obtaining  the  desired  capability  as  identified  by  the 
program’s  requirements  and  specifications  (711  HPW/HPO,  2009).  Employing  an  HSIP 
satisfies  DoD  Instruction  5000.02,  where  it  calls  for  the  program  manager  to  have  a  plan 
for  HSI  in  place  early  in  the  acquisition  process  to  optimize  total  system  performance, 
minimize  total  ownership  costs,  and  ensure  that  the  system  is  built  to  accommodate  the 
characteristics  of  the  user  population. 

The  HSIP  serves  as  an  evolutionary  formal  document  that  tracks  HSI  execution 
and  identifies  and  mitigates  HSI  risk  as  the  acquisition  program  progresses.  For  the 
present  study,  the  usefulness  of  the  HRL  framework  to  directly  support  this  effort  will  be 
evaluated.  Consideration  will  be  given  of  its  ability  to  provide  a  potentially  exhaustive 
list  of  candidate  activities  to  be  tailored  for  GMOD  to  populate  HSI  Planning  to  inform 
requirements,  acquisition,  and  sustainment  documents  of  record.  In  addition,  the  HRL 
will  be  evaluated  in  its  ability  to  provide  those  using  the  HSIP  a  risk  management  metric 
and  process  that  is  similar  to  the  TRL  metric  and  process,  but  that  explicitly  links 
technology  development  to  the  effective  design  and  specification  of  human-centered 
systems  in  Defense  Acquisition. 

B.  METHODOLOGY 

1.  Instrument 

For  the  present  study,  a  questionnaire  was  designed  to  gain  feedback  regarding 
the  HRL’s  ability  to  effectively  contribute  to  the  creation  and  sustainment  of  an 
acquisition  program’s  HSIP.  Consideration  was  given  to  conducting  a  semi-structured 
interview.  However,  it  was  decided  that,  given  the  complexity  of  the  HRL  framework,  a 
questionnaire  would  allow  the  SME  to  provide  more  thoughtful  feedback,  and  allow  him 
to  carry  out  the  evaluation  at  his  convenience.  The  questionnaire  contained  a  total  of  16 
items  separated  into  two  sections  as  shown  in  Appendix  H.  The  first  section  consisted  of 
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six  Likert  Scale  statements  that  focused  on  the  completeness  and  relevance  of  the  HSI 
activities  and  categories  listed  within  the  HRL  framework.  The  second  section  consisted 
of  1 0  orders  of  merit  ranking  items  that  were  designed  to  gain  feedback  as  to  the  relative 
value/importance  of  the  current  and  proposed  categories  contained  in  the  HRL 
framework  when  being  used  to  support  an  HSIP.  Within  both  sections,  the  opportunity 
for  comments  was  made  available. 

2.  Sample 

The  questionnaire  was  administered  to  the  chief  HSI  practitioner  working  within 
the  GMOD  acquisition  program.  As  a  senior  research  aerospace  engineer  and  lead  for 
HSI  transition  in  the  Air  Force  Human  Performance  Integration  Directorate,  this  SME 
was  recognized  to  have  extensive  knowledge  and  experience  in  developing  and  managing 
HSIPs.  Much  like  the  evaluation  that  took  place  in  Chapter  IV,  the  expert  served  as  a 
participant  to  the  research  and  was  not  subject  of  the  research.  Therefore,  the  Naval 
Postgraduate  School’s  Institutional  Review  Board  determined  the  study  protocol  did  not 
require  IRB  involvement 

C.  RESULTS  AND  DISCUSSION 

In  the  first  section  of  the  HRL  questionnaire,  data  were  gathered  based  on  the 
ratings  of  agreement  from  the  five-point  Likert  scale  ranging  from  “strongly  agree”  to 
“strongly  disagree.”  For  the  most  part,  there  was  a  general  agreement  that  the  HRL  could 
serve  as  a  beneficial  tool  in  the  development  of  an  acquisition  program’s  HSIP.  The 
following  statement  represented  the  typical  comments  made  by  the  case  study  SME: 

“ Agreed ...  the  HRL  provides  a  context  of  how  early  in  the  program  these 
activities  should  be  conducted.  Ties  to  risk  by  seeing  when  this  should  be  done  compared 
to  when  it  is  planned  to  be  done  (or  that  it  hasn  7  been  done).  ” 

Specifically,  the  expert  felt  the  milestone-sensitive  HSI  activities  listed  in  the 
HRL  matrix  could  provide  an  effective  framework  for  an  HSI  working  group  to  tailor  for 
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comprehensive  HSI  planning.  Additionally,  it  was  agreed  that  using  the  HRL  framework 
as  an  HSI  maturity  metric  can  benefit  HSI  risk  identification  and  management  within  the 
HSIP. 

The  areas  of  disagreement  were  few,  but  important  none  the  less.  Consistent  with 
findings  from  the  larger  review  in  the  previous  study,  the  expert  did  not  feel  the  HRL 
framework  contained  an  exhaustive  list  of  HSI  activities  to  account  for  all  program  needs. 
Specifically,  the  SME  felt  that  more  activities  would  be  necessary  to  ultimately  manage 
and  sustain  an  HSIP  throughout  a  program’s  life  cycle.  The  current  activities  were 
thought  to  be  a  good  start;  however,  as  this  tool  continues  to  mature  and  is  used  for  a 
diverse  variety  of  acquisition  programs,  more  activities  were  felt  to  be  needed.  The 
following  is  the  comment  made  by  the  SME  regarding  this  issue: 

“/  think  this  is  a  great  start,  but  just  by  the  limited  number  of  activities,  I  think 
that  there  are  other  activities  that  we  will  need  to  add  as  this  tool  is  maturing,  especially 
as  we  use  this  prototype  tool  for  a  diverse  variety  of  acquisition  programs.  ” 

Other  disagreement  involved  whether  the  proposed  categories  (i.e.,  column 
headings)  for  future  HRL  development  efforts  represented  a  complete  list  of  the  key  areas 
of  information  needed  to  perform  effective  HSI  Planning.  A  major  category  that  was 
stated  to  be  missing  was  some  linkage  to  which  HSI  domain  each  of  the  sub-activities 
pertained  to,  or  some  other  type  of  solution  (e.g.  color,  numbering  by  domain,  etc.)  that 
would  identify  the  domain.  The  domain  links,  as  recommended  by  the  expert,  could 
introduce  an  even  more  valuable  category  that  would  calculate  risk  levels  per  domain. 

In  the  second  section  of  the  HRL  validation,  data  collected  were  based  on  the 
order  of  value/importance  for  the  current  and  proposed  categories  contained  in  the  HRL 
framework  when  being  used  to  support  an  HSIP  (tied  values  of  ranking  were  allowed). 
The  HRL  category  that  simply  designated  the  HRL  level  (i.e.,  1,  1.1,  1.2,  etc.)  was 
considered  most  important,  followed  by  Products  (of  Activity),  Activity,  Sub-Activity 
Detail,  Target  Documents,  References  (tied  with  Target  Documents),  Action  OPR, 
Integration  Notes/Comments  (tied  with  Action  OPR),  Rough  Cost  Estimate,  and 
Acquisition  &  Sustainment  Toolkit  Linkages  (did  not  receive  a  value  ranking).  The  expert 
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stated  that  the  framework  categories  that  received  higher  rankings  (particularly,  HRL, 
Products  (of  Activity),  and  Activity)  was  due  to  their  ability  to  efficiently  tie  into  risk 
management  structures  and  not  only  provide  a  metric  on  which  to  base  developmental 
decisions,  but  they  also  provide  program  managers  with  visible  indicators  of  what  should 
be  done  compared  to  when  it  is  planned  to  be  done.  Lastly,  the  category  of  Acquisition  & 
Sustainment  Toolkit  Linkages  was  not  ranked  because  the  expert  did  not  have  enough 
familiarity  with  the  toolkit  to  pass  judgment  of  its  importance. 

D.  SUMMARY 

The  results  of  this  case  study  showed  that  the  responses  of  a  SME  considering  the 
usefulness  of  the  HRL  framework  with  reference  to  a  particular  acquisition  program  were 
consistent  with  findings  from  the  study  reported  in  the  previous  chapter.  The  SME 
thought  the  HRL  could  serve  as  a  beneficial  tool  in  the  development  of  an  acquisition 
program’s  HSIP.  The  HSI  practitioner  who  participated  in  this  effort  also  considered  the 
HRL  to  be  a  valuable  HSI  maturity  metric  that  could  benefit  HSI  risk  identification  and 
management  within  program  risk  management  structures.  In  regards  to  needed 
improvements,  the  SME  recommended  to  expand  the  HRL’s  activities  to  account  for  the 
diverse  variety  of  programs  that  exist  in  acquisition.  In  the  next  and  final  chapter, 
recommendations  will  be  made  on  improvements  that  need  to  be  made  to  the  HRL 
framework. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  SUMMARY 

This  thesis  described  the  initial  development  of  a  Human  Readiness  Level 
framework,  designed  to  complement  the  existing  Technology  Readiness  Level  currently 
being  used  within  the  Integrated  Defense  Acquisition,  Technology,  and  Logistics 
(IDAT&L)  Life  Cycle  Management  System.  Technology  maturity  assessment  tools,  such 
as  the  TRL,  serve  as  systematic  measurement  systems  that  are  used  as  entry  and  exit 
criteria  for  transitioning  milestones  and  are  integral  components  to  program  risk 
management  structures.  Yet,  the  TRL  has  proven  incapable  of  consistently  capturing  the 
human-related  aspects  of  technology  development  and  their  association  with  technology 
readiness. 

The  proposed  HRL  framework  was  developed  to  add  clarity  to  technical  readiness 
assessments  by  emphasizing  the  socio-technical  attributes  of  system  development. 
Specifically,  the  HRL  aims  to  reduce  technology  risks  related  to  the  human  element  by 
ensuring  the  adequate  incorporation  of  Human  Systems  Integration  during  technology 
maturity  evaluations  and  by  explicitly  linking  technology  development  to  the  effective 
design  and  specification  of  human-centered  systems  in  Defense  Acquisition.  The  HRL 
accomplishes  this  risk  reduction  by  providing  a  time  and  milestone-sensitive  roadmap  of 
activities  that:  a)  detail  critical  organizational  milestones  (that  ensure  functional 
commitment  to  HSI);  and  b)  define  HSI  domain-specific  data  collection  and  analysis,  and 
a  clear  process  for  their  management.  The  primary  measure  of  HSI  risk  and  maturity 
level  is  based  on  the  execution  and  outcome  of  those  milestone-sensitive  management 
and  analytical  activities  that  have  been  designated  in  each  progressive  HRL. 

To  evaluate  the  utility  of  the  initial  HRLs  with  the  user  population,  two  research 

efforts  were  carried  out.  In  the  first,  an  evaluation  questionnaire  designed  to  gain 

feedback  as  to  the  overall  worth  and  potential  usefulness  of  the  HRL  framework  was 

given  to  43  subject  matter  experts  in  the  fields  of  HSI  and  Defense  Acquisition.  A  total  of 

15  responses  were  obtained.  Results  of  this  evaluation  indicated  relative  agreement 
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among  SMEs  that  the  first  iteration  of  the  HRL  concept,  once  fully  developed,  will  serve 
as  a  valuable  HSI  planning  tool  and  complementary  (HSI-specific)  metric  in  program  risk 
management  structures.  In  order  to  improve  the  framework,  many  SMEs  recommended 
expanding  the  HRL’s  list  of  activities  to  account  for  a  more  diverse  variety  of  programs 
that  exist  in  acquisition. 

The  second  research  effort  was  a  case  study  regarding  the  usefulness  of  the  HRL 
framework  when  applied  to  a  specific  acquisition  program.  Specifically,  the  HRL  was 
evaluated  in  its  ability  to  effectively  contribute  to  the  creation  and  sustainment  of  the 
acquisition  program’s  HSI  Plan  for  the  Ground  Control  Station  Modernization.  Results  of 
this  SME  evaluation  suggested  a  general  agreement  that  the  HRL  framework  could  serve 
as  a  beneficial  tool  in  the  development  of  an  acquisition  program’s  comprehensive  HSIP. 
Consistent  with  findings  from  the  previous  evaluation,  the  SME  did  not  feel  that  the  HRL 
framework  contained  all  of  the  necessary  HSI  activities  to  account  for  all  program  needs. 
The  feedback  and  lessons  learned  from  both  studies  were  documented  and  will  form  the 
basis  for  future  HRL  development  efforts.  Recommendations  for  improvements  to  the 
HRL  framework  are  provided  in  the  next  section. 

B.  RECOMMENDATIONS  AND  FURTHER  RESEARCH 

Areas  of  particular  importance  that  need  to  be  addressed  in  the  next  iteration  of 
the  HRL  framework  are: 

1.  Link  HRLs  to  Cost  Estimation  Processes 

In  the  context  of  a  competitive  cost,  schedule,  and  performance-driven  acquisition 
enterprise,  accurately  funding  HSI  initiatives  continues  to  be  a  critical  component.  Often, 
program  requirements  are  inappropriately  funded  because  the  costs  involving  HSI- 
specific  data  collection  and  analysis  were  not  estimated  in  advance.  Such  exclusions 
produce  cost  overruns,  schedule  delays,  and  deficiencies  in  performance  that  have 
considerable  impact  on  the  eventual  total  life  cycle  cost.  Therefore,  continued  research 
must  focus  on  linking  HRLs  to  acquisition  cost  estimation  processes.  Specifically,  efforts 
should  concentrate  on  detailing  how  HRLs  establish  a  basis  for  defining  HSI-specific  cost 
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estimation  factors  and  elements  paralleling  Work  Breakdown  Structure  links  to  system 
components.  This  will  effectively  “fund  the  requirement”  by  directly  informing  Planning, 
Programming,  Budgeting,  and  Execution  processes  (employed  by  DoD  for  strategic 
planning  and  resource  allocation). 

2.  Standardize  HRL  Metrics  Within  Technology  Transition 

The  DoD  S&T  community  is  tasked  with  ensuring  that  technologies  are  mature 
when  DoD’s  acquisition  community  takes  over  and  integrates  the  technologies  into 
weapon  systems.  This  transition  involves  management  and  funding  responsibilities  to 
gradually  shift  from  the  lab  to  the  product  line.  In  order  to  support  this  transition  with 
significant  HSI  influence,  continued  research  should  focus  on  connecting  HRLs  to 
Technology  Transition  Agreements.  Specifically,  efforts  should  define  how  HRLs 
articulate  external  dependencies  on  technology  base  projects  and  provide  exit  criteria  for 
the  technology  to  transition  into  the  acquisition  program  (i.e.,  HSI  metrics).  This  will 
provide  S&T  and  acquisition  program  managers  with  demonstrated  knowledge  about  the 
human-specific  readiness  and  the  potential  risks  of  including  the  technology  on  a 
weapons  program. 

3.  Define  HRL  Assessment  Procedures 

A  standard  repeatable  method  for  determining  the  HRL  achieved  by  a  given 
technology  must  exist  if  the  HRL  is  to  be  effectively  used  and  consistently  trusted  within 
Defense  Acquisition.  Therefore,  in  a  later  stage  of  HRL  development,  a  fonnal, 
systematic,  and  explicit  process  should  be  created  for  risk  management  and  HRL 
reporting.  The  process  will  need  to  define  how  a  measure  is  obtained  and  how  to  assign 
each  HRL  as  an  assessed  program  achievement  value  and  risk  management  metric. 
Ultimately,  this  will  be  a  vital  step  towards  joining  TRLs  in  program  risk  management 
structures  within  DoD. 
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c. 


CONCLUSION 


Although  HSI  is  a  fundamental  component  of  a  total  systems  approach,  the 
successful  integration  of  HSI  into  systems  engineering  and  acquisition  life  cycles 
continues  to  be  a  challenge.  Methodologies  and  tools,  such  as  the  HRL  framework,  are 
needed  to  enhance  human/system  perfonnance,  maximize  operational  effectiveness,  and 
prevent  cost  overruns.  It  is  recognized  that  further  development  and  evaluation  of  the 
HRL  concept  is  required.  However,  the  initial  framework  presented  in  this  thesis  takes 
that  first  step  towards  providing  acquisition  professionals  a  comprehensive  guide  that 
ensures  human-centric  priorities  are  addressed  throughout  all  phases  and  milestones  of 
Defense  Acquisition. 
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APPENDIX  A.  HARDWARE  TRL  DEFINITIONS,  DESCRIPTIONS, 
AND  SUPPORTING  INFORMATION 


Hardware  TRL  Definitions,  Descriptions,  and  Supporting  Information  (DoD,  2009) 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles 
observed  and 
reported. 

Lowest  level  of  technology 
readiness.  Scientific 
research  begins  to  be 
translated  into  applied 
research  and  development 
(R&D).  Examples  might 
include  paper  studies  of  a 
technology's  basic 
properties. 

Published  research  that  identifies  the 
principles  that  underlie  this  technology. 
References  to  who,  where,  when. 

2 

Technology  con¬ 
cept  and/or  appli¬ 
cation  formulated. 

Invention  begins.  Once 
basic  principles  are 
observed,  practical  applica¬ 
tions  can  be  invented.  Appli¬ 
cations  are  speculative,  and 
there  may  be  no  proof  or 
detailed  analysis  to  support 
the  assumptions.  Examples 
are  limited  to  analytic 
studies. 

Publications  or  other  references  that  out¬ 
line  the  application  being  considered  and 
that  provide  analysis  to  support  the 
concept. 

3 

Analytical  and 
experimental  criti¬ 
cal  function  and/or 
characteristic  proof 
of  concept. 

Active  R&D  is  initiated.  This 
includes  analytical  studies 
and  laboratory  studies  to 
physically  validate  the 
analytical  predictions  of 
separate  elements  of  the 
technology.  Examples 
include  components  that  are 
not  yet  integrated  or 
representative. 

Results  of  laboratory  tests  performed  to 
measure  parameters  of  interest  and  com¬ 
parison  to  analytical  predictions  for  critical 
subsystems.  References  to  who,  where, 
and  when  these  tests  and  comparisons 
were  performed. 

4 

Component  and/or 
breadboard  valida¬ 
tion  in  a  laboratory 
environment. 

Basic  technological  compo¬ 
nents  are  integrated  to 
establish  that  they  will  work 
together.  This  is  relatively 
“low  fidelity”  compared  with 
the  eventual  system.  Exam¬ 
ples  include  integration  of 
'ad  hoc"  hardware  in  the 
laboratory. 

System  concepts  that  have  been  consi¬ 
dered  and  results  from  testing  laboratory- 
scale  breadboard(s).  References  to  who 
did  this  work  and  when.  Provide  an  esti¬ 
mate  of  how  breadboard  hardware  and 
test  results  differ  from  the  expected  sys¬ 
tem  goals. 

5 

Component  and/or 
breadboard  valida¬ 
tion  in  a  relevant 
environment. 

Fidelity  of  breadboard 
technology  increases 
significantly.  The  basic 
technological  components 
are  integrated  with  reason¬ 
ably  realistic  supporting 
elements  so  they  can  be 
tested  in  a  simulated  envi¬ 
ronment.  Examples  include 
“high-fidelity’’  laboratory 
integration  of  components. 

Results  from  testing  a  laboratory  bread¬ 
board  system  are  integrated  with  other 
supporting  elements  in  a  simulated  oper¬ 
ational  environment.  How  does  the  “rele¬ 
vant  environment”  differ  from  the 
expected  operational  environment?  How 
do  the  test  results  compare  with  expecta¬ 
tions?  What  problems,  if  any,  were 
encountered?  Was  the  breadboard  sys¬ 
tem  refined  to  more  nearly  match  the 
expected  system  goals? 
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6 

System/subsystem 
model  or  prototype 
demonstration  in  a 
relevant 
environment. 

Representative  model  or 
prototype  system,  which  is 
well  beyond  that  of  TRL  5,  is 
tested  in  a  relevant  environ¬ 
ment.  Represents  a  major 
step  up  in  a  technology's 
demonstrated  readiness. 
Examples  include  testing  a 
prototype  in  a  high-fidelity 
laboratory  environment  or  in 
a  simulated  operational 
environment. 

Results  from  laboratory  testing  of  a  proto¬ 
type  system  that  is  near  the  desired  con¬ 
figuration  in  terms  of  performance,  weight, 
and  volume.  How  did  the  test  environment 
differ  from  the  operational  environment? 
Who  performed  the  tests?  How  did  the 
test  compare  with  expectations?  What 
problems,  if  any,  were  encountered? 

What  are/were  the  plans,  options,  or 
actions  to  resolve  problems  before 
moving  to  the  next  level? 

7 

System  prototype 
demonstration  in 
an  operational 
environment. 

Prototype  near  or  at  planned 
operational  system.  Repre¬ 
sents  a  major  step  up  from 
TRL  6  by  requiring  demon¬ 
stration  of  an  actual  system 
prototype  in  an  operational 
environment  (e.g..  in  an  air¬ 
craft.  in  a  vehicle,  or  in 
space). 

Results  from  testing  a  prototype  system  in 
an  operational  environment.  Who  per¬ 
formed  the  tests?  How  did  the  test  com¬ 
pare  with  expectations?  What  problems, 
if  any,  were  encountered?  What  are/were 
the  plans,  options,  or  actions  to  resolve 
problems  before  moving  to  the  next  level? 

8 

Actual  system 
completed  and 
qualified  through 
test  and 
demonstration. 

Technology  has  been 
proven  to  work  in  its  final 
form  and  under  expected 
conditions.  In  almost  all 
cases,  this  TRL  represents 
the  end  of  true  system 
development.  Examples 
include  developmental  test 
and  evaluation  (DT&E)  of 
the  system  in  its  intended 
weapon  system  to  deter¬ 
mine  if  it  meets  design 
specifications. 

Results  of  testing  the  system  in  its  final 
configuration  under  the  expected  range  of 
environmental  conditions  in  which  it  will 
be  expected  to  operate.  Assessment  of 
whether  it  will  meet  its  operational 
requirements.  What  problems,  if  any, 
were  encountered?  What  are/were  the 
plans,  options,  or  actions  to  resolve 
problems  before  finalizing  the  design? 

9 

Actual  system 
proven  through 
successful  mission 
operations. 

Actual  application  of  the 
technology  in  its  final  form 
and  under  mission  condi¬ 
tions,  such  as  those 
encountered  in  operational 
test  and  evaluation  (OT&E). 
Examples  include  using  the 
system  under  operational 
mission  conditions. 

OT&E  reports. 
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APPENDIX  B.  SOFTWARE  TRL  DEFINITIONS,  DESCRIPTIONS, 
AND  SUPPORTING  INFORMATION 


Software  TRL  Definitions,  Descriptions,  and  Supporting  Information  (DoD,  2009) 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles 
observed  and 
reported. 

Lowest  level  of  software 
technology  readiness.  A 
new  software  domain  is 
being  investigated  by  the 
basic  research  community. 
This  level  extends  to  the 
development  of  basic  use, 
basic  properties  of  software 
architecture,  mathematical 
formulations,  and  general 
algorithms. 

Basic  research  activities,  research 
articles,  peer-reviewed  white  papers, 
point  papers,  early  lab  model  of  basic 
concept  may  be  useful  for  substantiating 
the  TRL. 

2 

Technology  con¬ 
cept  and/or  appli¬ 
cation  formulated. 

Once  basic  principles  are 
observed,  practical  applica¬ 
tions  can  be  invented. 
Applications  are  speculative, 
and  there  may  be  no  proof 
or  detailed  analysis  to  sup¬ 
port  the  assumptions. 
Examples  are  limited  to 
analytic  studies  using  syn¬ 
thetic  data. 

Applied  research  activities,  analytic  stu¬ 
dies,  small  code  units,  and  papers  com¬ 
paring  competing  technologies. 

3 

Analytical  and 
experimental  criti¬ 
cal  function  and/or 
characteristic  proof 
of  concept. 

Active  R&D  is  initiated.  The 
level  at  which  scientific  fea¬ 
sibility  is  demonstrated 
through  analytical  and  labor¬ 
atory  studies.  This  level 
extends  to  the  development 
of  limited  functionality  envi¬ 
ronments  to  validate  critical 
properties  and  analytical 
predictions  using  non-inte- 
grated  software  components 
and  partially  representative 
data. 

Algorithms  run  on  a  surrogate  processor 
in  a  laboratory  environment,  instrumented 
components  operating  in  a  laboratory 
environment,  laboratory  results  showing 
validation  of  critical  properties. 

4 

Module  and/or 
subsystem  valida¬ 
tion  in  a  laboratory 
environment  (i.e., 
software  prototype 
development 
environment). 

Basic  software  components 
are  integrated  to  establish 
that  they  will  work  together. 
They  are  relatively  primitive 
with  regard  to  efficiency  and 
robustness  compared  with 
the  eventual  system.  Archi¬ 
tecture  development 
initiated  to  include  interope¬ 
rability,  reliability,  maintain¬ 
ability,  extensibility, 
scalability,  and  security 
issues.  Emulation  with  cur¬ 
rent/legacy  elements  as 
appropriate.  Prototypes 
developed  to  demonstrate 
different  aspects  of  eventual 
system. 

Advanced  technology  development, 
stand-alone  prototype  solving  a  synthetic 
full-scale  problem,  or  standalone  proto¬ 
type  processing  fully  representative  data 
sets. 
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TRL 

Definition 

Description 

Supporting  Information 

5 

Module  and/or 
subsystem  valida¬ 
tion  in  a  relevant 
environment. 

Level  at  which  software 
technology  is  ready  to  start 
integration  with  existing 
systems.  The  prototype 
implementations  conform  to 
target  environment/inter¬ 
faces.  Experiments  with 
realistic  problems.  Simu¬ 
lated  interfaces  to  existing 
systems.  System  software 
architecture  established. 
Algorithms  run  on  a  proces¬ 
sors)  with  characteristics 
expected  in  the  operational 
environment. 

System  architecture  diagram  around 
technology  element  with  critical  perfor¬ 
mance  requirements  defined.  Processor 
selection  analysis,  Simulation/Stimulation 
(Sim/Stim)  Laboratory  buildup  plan.  Soft¬ 
ware  placed  under  configuration  manage¬ 
ment.  Commercial-of-the-shelf/ 
government-off-the-shelf  (COTS/GOTS) 
components  in  the  system  software 
architecture  are  identified. 

6 

Module  and/or 
subsystem  valida¬ 
tion  in  a  relevant 
end-to-end 
environment. 

Level  at  which  the  engi¬ 
neering  feasibility  of  a 
software  technology  is 
demonstrated.  This  level 
extends  to  laboratory  proto¬ 
type  implementations  on 
full-scale  realistic  problems 
in  which  the  software 
technology  is  partially  inte¬ 
grated  with  existing  hard¬ 
ware/software  systems. 

Results  from  laboratory  testing  of  a  proto¬ 
type  package  that  is  near  the  desired 
configuration  in  terms  of  performance, 
including  physical,  logical,  data,  and  secu¬ 
rity  interfaces.  Comparisons  between 
tested  environment  and  operational  envi¬ 
ronment  analytically  understood.  Analysis 
and  test  measurements  quantifying  con¬ 
tribution  to  system-wide  requirements 
such  as  throughput,  scalability,  and  relia¬ 
bility.  Analysis  of  human-computer  (user 
environment)  begun. 

7 

System  prototype 
demonstration  in 
an  operational, 
high-fidelity 
environment. 

Level  at  which  the  program 
feasibility  of  a  software 
technology  is  demonstrated. 
This  level  extends  to  opera¬ 
tional  environment  prototype 
implementations,  where 
critical  technical  risk  functio¬ 
nality  is  available  for  dem¬ 
onstration  and  a  test  in 
which  the  software  technol¬ 
ogy  is  well  integrated  with 
operational  hardware/soft¬ 
ware  systems. 

Critical  technological  properties  are 
measured  against  requirements  in  an 
operational  environment. 

8 

Actual  system 
completed  and 
mission  qualified 
through  test  and 
demonstration  in 
an  operational 
environment. 

Level  at  which  a  software 
technology  is  fully  integrated 
with  operational  hardware 
and  software  systems. 
Software  development 
documentation  is  complete. 

All  functionality  tested  in 
simulated  and  operational 
scenarios. 

Published  documentation  and  product 
technology  refresh  build  schedule.  Soft¬ 
ware  resource  reserve  measured  and 
tracked. 

9 

Actual  system 
proven  through 
successful  mission- 
proven  operational 
capabilities. 

Level  at  which  a  software 
technology  is  readily 
repeatable  and  reusable. 

The  software  based  on  the 
technology  is  fully  integrated 
with  operational  hardware/ 
software  systems.  All  soft¬ 
ware  documentation  veri¬ 
fied.  Successful  operational 
experience.  Sustaining 
software  engineering  sup¬ 
port  in  place.  Actual  system. 

Production  configuration  management 
reports.  Technology  integrated  into  a 
reuse  "wizard." 
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APPENDIX  C.  MANAGEMENT  OF  TECHNOLOGY  MATURITY  IN 
THE  DEFENSE  ACQUISITION  MANAGEMENT  SYSTEM 


The  Defense  Acquisition  Management  System,  as  stated  in  the  2009  Defense 
Acquisition  Guidebook  (DAG),  exists  to  manage  the  Nation's  investments  in  technologies, 
programs,  and  product  support  necessary  to  achieve  the  National  Security  Strategy  and 
support  the  United  States  Armed  Forces.  More  specifically,  it  establishes  a  simplified  and 
flexible  management  framework  for  translating  capability  needs  and  technology 
opportunities,  based  on  approved  capability  needs,  into  stable,  affordable,  and  well- 
managed  acquisition  programs  that  include  weapon  systems,  services,  and  automated 
information  systems  (AISs)  (DoDI  5000.02,  2008). 
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Figure  4.  Integrated  Defense  Acquisition,  Technology  and  Logistics  Life  Cycle 
Management  System  (After  DoDI  5000.02,  2008) 


Figure  4  depicts  the  evolution  of  technology  in  the  DoD  Acquisition  Life  Cycle. 
This  process,  beginning  with  User  Needs  and  Technology  Opportunities,  uses  Joint 
Concepts,  integrated  architectures,  and  an  analysis  of  doctrine,  organization,  training, 
materiel,  leadership  and  education,  personnel,  and  facilities  (DOTMLPF)  to  define 
needed  capabilities  and  to  guide  the  development  of  affordable  systems  through  five 
subsequent  phases: 


67 


1 .  Materiel  Solution  Analysis  (MSA) 

2.  Technology  Development  (TD) 

3.  Engineering  and  Manufacturing  Development  (EMD) 

4.  Production  and  Deployment  (PD) 

5.  Operations  and  Support  (OS) 

Of  the  five  phases,  the  first  four  place  emphasis  on  technology  and  product 
development.  During  these  early  stages  a  new  technology  undergoes  research  and 
development,  integration  into  systems,  and  finally  the  manufacture  and  distribution  in 
weapon  system  form.  The  last  phase,  Operations  and  Support,  attends  to  the  system  after 
it  has  been  fielded  and  represents  the  majority  of  the  actual  time  a  system  and  its 
integrated  technology  must  be  managed  (Nolte,  2008). 

In  DoD  acquisition,  a  crucial  part  of  overall  program  management  is  the 
management  of  technology  maturity  and  mitigation  of  technology  integration  risk. 
Therefore,  per  DoD  Instruction  5000.02  Operation  of  the  Defense  Acquisition  System, 
objective  evaluations  of  technology  maturity  are  required  to  be  routine  aspects 
throughout  the  entire  acquisition  life  cycle.  The  evaluation,  officially  referred  to  by  DoD 
as  the  Technology  Readiness  Assessment  (TRA),  is  the  formal,  systematic,  metrics-based 
process  used  to  assess  the  maturity  of  critical  hardware  and  software  technologies  to  be 
used  in  DoD  systems.  The  TRA  is  conducted  by  an  Independent  Review  Team  (IRT)  of 
subject  matter  experts  (SMEs)  who  use  the  TRL  as  the  metric  to  assess  technology 
maturity. 

All  DoD  acquisition  programs  going  through  the  Defense  Acquisition  System 
must  have  a  formal  TRA  completed  at  Milestone  B,  which  is  normally  the  formal 
initiation  of  an  acquisition  program,  and  also  at  Milestone  C.  However,  assessments  of 
technology  readiness  or  TRA-like  activities  other  than  the  formal  TRAs  at  Milestones  B 
and  C  take  place  over  the  acquisition  life  cycle. 

Using  infonnation  and  guidance  found  in  DoD’s  2009  Technology  Readiness 
Assessment  Deskbook,  the  following  paragraphs  summarize  how  the  knowledge 


68 


concerning  technology  maturity  evolves  over  time  within  acquisition  and  discusses  the 
mandated  TRL  levels  that  must  be  met  to  progress  through  the  acquisition  life  cycle. 

A.  EARLY  EVALUATIONS  OF  TECHNOLOGY  MATURITY 

In  the  Materiel  Solution  Analysis  phase,  an  Analysis  of  Alternatives  (AoA)  is 
conducted  to  identify  potential  materiel  solutions,  based  on  a  cost-benefit  analysis.  While 
this  is  occurring,  early  Systems  Engineering  (SE)  activities,  such  as  the  proposed 
Engineering  Analysis  of  Potential  System  Solutions,  are  conducted.  The  possible  materiel 
solutions  are  then  expected  to  undergo  an  Early  Evaluation  of  Technological  Maturity, 
provided  sufficient  technical  information  exists  to  support  such  an  evaluation.  This 
evaluation  is  an  activity  separate  from  the  formal  TRA  and  is  conducted  shortly  before 
Milestone  A. 

This  body  of  work — the  AoA,  the  early  SE,  and  the  Early  Evaluation  of 
Technological  Maturity — forms  the  basis  of  the  Technology  Development  Strategy 
(TDS)  for  evaluating  the  technology  options  in  the  materiel  solution  to  the  capability 
need  identified  in  the  approved  Initial  Capabilities  Document  (ICD).  DoD  relies  on  the 
TDS  to  show  how  the  technologies  (those  known  by  Milestone  A  to  be  critical  for  the 
successful  realization  of  the  chosen  materiel  solution)  will  be  demonstrated  in  a  relevant 
environment  (equivalent  to  TRL  6)  before  they  are  used  in  Engineering  and 
Manufacturing  Development.  If  the  AoA  and  early  SE  work  does  not  result  in  sufficient 
technical  infonnation  to  support  adequate  evaluation  of  technology  maturity,  acquisition 
programs  are  still  expected  to  have  an  evaluation  completed  prior  to  Milestone  A  so  that 
critical  technologies  can  be  matured  during  the  subsequent  Technology  Development 
phase. 

The  TRA  Deskbook  states  a  recommended  best  practice  is  to  use  the  results  of 
this  early  evaluation  of  technology  maturity  as  follows: 

•  To  provide  a  basis  for  modifying  the  requirements  if  technological  risks 
are  too  high. 
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•  To  support  the  development  of  Technology  Maturity  Plans  (TMPs)  that 
show  how  all  likely  critical  technologies  will  be  demonstrated  in  a 
relevant  environment  before  preliminary  design  begins  at  the  full  system 
level. 

•  To  refine  the  TDS. 

•  To  inform  the  test  and  evaluation  (T&E)  community  about  technology 
maturity  needs. 

•  To  ensure  that  all  potential  critical  technologies  are  included  in  the 
program’s  risk  management  database  and  plan. 

•  To  establish  Technology  Transition  Agreements  (TTAs)  to  articulate 
external  dependencies  on  technology  base  projects  and  to  define  the 
specific  technologies,  technology  demonstration  events,  and  exit  criteria 
for  the  technology  to  transition  into  the  acquisition  program. 

Ultimately,  the  early  evaluation  of  technology  maturity  conducted  at  or  shortly 
before  Milestone  A  helps  evaluate  technology  alternatives  and  risks  and,  by  doing  so, 
helps  the  acquisition  Program  Manager  (PM)  refine  the  plans  for  achieving  mature 
technologies  before  Milestone  B  approval  is  sought.  However,  it  is  worth  noting  that  this 
early  evaluation  of  technology  maturity  is  not  a  replacement  for  the  Milestone  B  TRA 
(DoD,  2009). 

B.  MILESTONE  B  TRA 

Throughout  the  beginning  MSA  and  Technology  Development  phases,  the  DoD 
science  and  technology  (S&T)  community  is  tasked  with  ensuring  that  technologies  are 
mature  when  DoD’s  acquisition  community  takes  over  and  integrates  the  technologies 
into  weapon  systems.  This  transition  occurs  at  Milestone  B  which  begins  the  EMD  phase 
and,  as  mentioned  earlier,  normally  marks  the  fonnal  initiation  of  an  acquisition  program. 
To  avoid  programs  from  entering  the  EMD  phase  of  the  Defense  Acquisition  System 
with  immature  technologies,  Title  10  United  States  Code  (U.S.C.)  Section  2366b 
requires,  in  part,  that  the  Milestone  Decision  Authority  (MDA)  certify  that  the  technology 
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in  Major  Defense  Acquisition  Programs  (MDAPs),  including  space  MDAPs,  has  been 
demonstrated  in  a  relevant  environment  (TRL  6)  before  Milestone  B  approval.  In 
addition,  while  10  U.S.C.  2366b  is  only  binding  for  MDAPs,  the  DoD  is  also  requiring 
Major  Automated  Information  System  (MAIS)  acquisitions  to  meet  the  same  technology 
maturity  standard  at  Milestone  B.  The  formal  TRA  process  and  report  serves  as  the  basis 
of  the  MDA  certification. 

In  cases  where  the  technology  in  the  program  has  not  been  demonstrated  in  a 
relevant  environment,  the  MDA  is  allowed  to  waive  the  certification  requirement  if  it 
determines  that  such  a  requirement  would  hinder  the  DoD’s  ability  to  meet  critical 
national  security  objectives.  However,  as  a  matter  of  practice,  such  waivers  will  only  be 
granted  in  extraordinary  circumstances.  In  fact,  DoDI  5000.02  requires  Request  for 
Proposal  (RFP)  language  that  prevents  the  award  of  an  EMD  contract  if  it  includes 
technologies  that  have  not  been  demonstrated  to  be  mature.  Therefore,  a  generic  TRA  not 
based  on  a  planned  technical  solution  is  not  acceptable  by  DoD.  The  TRA  must  be  based 
on  the  technologies  in  the  system  and  must  be  perfonned  on  all  the  competitors’ 
proposals  in  a  source  selection. 

C.  MILESTONE  C  TRA 

Milestone  C  marks  approval  to  enter  Low-Rate  Initial  Production  (LRIP)  for 
hardware  systems  and  limited  deployment  in  support  of  operational  testing  for  MAIS 
programs  or  for  software-intensive  systems  that  have  no  production  components.  TRL  7 
or  higher  is  the  expected  state  of  technology  maturity  at  Milestone  C. 

The  Milestone  C  TRA  is  conducted  to  reflect  the  resolution  of  any  technology 
deficiencies  that  arose  during  EMD  and  it  serves  as  a  check  that  all  critical  technologies 
are  maturing  as  planned.  By  the  time  an  acquisition  program  reaches  Milestone  C,  all 
critical  technologies  are  expected  to  have  appropriate  advancements  while  continuing  to 
mature  through  established  TMPs.  Any  new  technologies  that  have  emerged  should  be 
identified,  and  their  maturation  plans  reviewed. 
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For  software,  TRL  7  means  that  all  source  codes  have  been  written  and  tested — 
not  only  as  an  independent  module  and/or  component,  but  also  as  integrated  into  the 
whole  system.  The  TRA  at  Milestone  C  is  important  for  MAIS  programs  because  it 

•  Documents  successful  developmental  test  and  evaluation  (DT&E). 

•  Examines  plans  for  maintenance  and  upgrades  to  ensure  that  no  new 
technologies  are  involved. 

•  Determines  whether  algorithms  will  transfer  successfully  when  host 
platforms  are  moved  and  full-scale  applications  are  initiated  in  a  real 
operational  environment. 

•  Identifies  where  new  Milestone  B  reviews  are  needed  for  future  releases  to 
initiate  efforts  to  improve  performance  and  determines  the  architectural 
changes  necessary  to  support  these  future  releases. 
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APPENDIX  D.  IMPACT  OF  IMMATURE  TECHNOLOGY  IN 
DEFENSE  ACQUISITION  PROGRAMS 


In  numerous  reports  to  Congressional  Committees,  the  Government 
Accountability  Office  (GAO)  has  addressed  the  problems  with  proceeding  in  system 
development  with  immature  technologies  and  has  detailed  the  resulting  cost  overruns, 
schedule  delays,  and  performance  shortfalls  that  have  been  undermining  DoD’s  buying 
power.  According  to  the  GAO,  this  dilemma  is  due  in  part  to  DoD’s  difficulty 
transitioning  technologies  from  a  technology  development  environment  to  an  acquisition 
program.  Because  the  acquisition  community  frequently  pulls  technologies  too  early,  it 
takes  on  the  additional  task  of  maturing  the  technologies — an  activity  that  is  the  primary 
responsibility  of  technology  developers — at  the  start  of  an  acquisition  program.  They 
further  state  that  the  start  of  a  program  ushers  in  a  high-cost,  delivery-oriented  phase  in 
which  the  acquisition  community  is  supposed  to  be  focused  on  integrating  subsystems 
and  working  on  system  development  and  demonstration.  DoD  has  continued  to  allow  the 
acquisition  community  to  take  over  this  task  before  the  S&T  community  considers  the 
technologies  ready  for  transition  (GAO,  2006).  The  lower  the  level  of  technology 
readiness,  the  more  ground  must  be  covered  to  bring  the  technology  to  the  point  at  which 
it  can  meet  the  intended  product’s  cost,  schedule,  and  performance  requirements  with 
little  risk.  Figure  5  illustrates  this  idea  on  the  following  page. 
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Figure  5.  Using  TRLs  to  Match  Technology  With  Product  Launch  Requirements 

(From  GAO,  1999) 

The  gap  between  the  maturity  of  the  technology  and  the  product’s  requirements 
represents  the  risks  or  unknowns  about  the  technology.  As  each  succeeding  level  of 
readiness  is  demonstrated,  unknowns  are  replaced  by  knowledge  and  the  gap  becomes 
smaller.  Ideally,  the  gap  is  closed  before  a  new  technology  is  included  in  a  new  product’s 
design  (GAO,  1999). 

Large,  highly  visible  acquisition  programs  are  by  no  means  immune  to  the 
problems  surrounding  immature  technology.  The  Joint  Strike  Fighter  (JSF),  for  example, 
is  the  most  expensive  aircraft  program  in  DoD  and  has  endured  multiple  acquisition 
schedule  changes  in  order  to  reduce  risk.  The  following  GAO  statement  found  in  the 
2001  Joint  Strike  Fighter  Acquisition  report,  illustrates  how  not  meeting  anticipated 
technology  maturity  can  delay  entry  into  key  development  phases: 

“Last  year,  we  testified  and  reported  that  a  key  objective  of  the  program’s 
acquisition  strategy  is  affordability  and  that  a  part  of  that  strategy —  entering  into 
engineering  and  manufacturing  development  with  low  technical  risk — would  not 
be  achieved  because  technologies  critical  to  meeting  the  program ’s  cost  and 
requirement  objectives  were  projected  to  be  at  low  levels  of  technical  maturity  in 
April  2001,  the  date  then  scheduled  for  awarding  the  engineering  and 
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manufacturing  development  contract.  We  stated  that  the  program ’s  approach  was 
not  consistent  with  best  practices  in  which  technologies  are  more  fully  developed 
before  proceeding  into  product  development...  Because  of  concerns  about  the 
adequacy  of  the  Joint  Strike  Fighter ’s  short  take-off  and  vertical  landing  flight 
test  program,  the  maturity  of  its  critical  technologies,  and  other  factors,  the 
Fiscal  Year  2001  National  Defense  Authorization  Act  directed  that  the  contract 
for  the  aircraft’s  engineering  and  manufacturing  development  not  be  awarded 
until  certain  criteria  were  met”  (GAO,  2001,  p.  1). 


The  impact  of  immature  technology  can  also  be  seen  in  the  GAO’s  2006  review 
of  52  major  DoD  weapon  programs.  In  this  review,  90  percent  of  the  programs  were 
found  to  have  started  with  immature  technologies  and  more  than  half  of  the  programs 
were  working  with  immature  technologies  at  design  review,  the  time  when  DoD 
acquisition  policy  expects  the  design  to  be  stable.  By  the  time  production  began,  one- 
third  of  the  programs  still  did  not  have  mature  technologies.  Not  surprisingly,  the  GAO 
found  that  DoD  research,  development,  test,  and  evaluation  cost  estimates  increased 
dramatically  for  programs  having  immature  technologies  at  program  start  (GAO,  2006). 
Figure  6  shows  the  average  cost  growth  of  DoD  programs  reviewed  when  technologies 
were  mature  and  immature  at  program  start. 


Maturity  level 


Figure  6.  Average  Program  Research,  Development,  Test,  and  Evaluation  Cost  Growth 
From  First  Full  Estimate  (sample  of  52  DoD  weapon  programs) 

(From  GAO,  2006) 


Programs  that  started  with  mature  technologies  averaged  a  fairly  small  4.8  percent 


cost  growth  above  the  first  full  estimate,  whereas  programs  that  started  with  immature 
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technologies  averaged  about  35  percent  cost  growth.  This  includes  some  programs  that 
experienced  significantly  greater  cost  growth.  A  consequence  of  this  cost  growth  is  that 
DoD  typically  delivers  weapon  systems  late,  reduces  quantities  to  stay  within  cost 
estimates,  shifts  funds  away  from  other  projects  to  make  up  for  added  costs,  or  some 
combination  of  the  three  (GAO,  2006). 
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APPENDIX  E.  THE  NATO  HUMAN  VIEW 


All  of  the  information  contained  in  this  appendix  was  derived  from  the  NATO 
Human  View  Handbook,  which  was  prepared  by  the  NATO  RTO  HFM-155  Human 
View  Workshop.  The  purpose  of  a  human  view  is  to  capture  the  human  requirements  and 
to  inform  on  how  humans  interact  with  systems.  The  Workshop  panel’s  final  set  of 
products  that  composes  the  NATO  Human  View  is  listed  below: 

HV-A  Concept 

HV-B  Constraints 

HV-C  Tasks 

HV-D  Roles 

HV-E  Human  Network 

HV-F  Training 

HV-G  Metrics 

HV-H  Human  Dynamics 


HV-A  CONCEPT 

The  HV-A  is  a  conceptual,  high-level  representation  of  the  human  component  of 
the  enterprise  architecture  framework.  Its  purpose  is  to  visualize  and  facilitate 
understanding  of  the  human  dimension  in  relation  to  operational  demands  and  system 
components.  It  serves  as  both  the  single  point  of  reference  and  departure  to  depict  how 
the  human  will  impact  perfonnance  (mission  success,  survivability,  supportability,  and 
cost)  and  how  the  human  will  be  impacted  by  the  system  design  and  operational  context 
(personnel  availability,  skill  demands,  training  requirements,  workload  and  well-being). 
The  HV-A  has  a  close  relationship  with  other  architecture  products  that  provide  high- 
level  concepts. 
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The  elements  of  the  HV-A  may  include: 

•  Pictorial  depictions  of  the  system  and  its  human  component 

•  High  level  indicators  of  where  human  system  interactions  may  occur 

•  A  textual  description  of  the  overall  human  component  of  the  system 

•  A  use  case  which  describes  the  human  process 

HV-B  CONSTRAINTS 

Not  only  is  the  human  the  most  important  and  unique  system  within  the  system- 
of-systems,  but  it  can  also  be  the  weakest  link  or  highest  risk  in  that  system.  Therefore 
expressing  the  capabilities  and  limitations  of  the  human  in  the  system  is  required.  The 
HV-B  contains  the  set  of  data  elements  that  are  used  to  adjust  the  expected  roles  and 
tasks.  It  acts  as  a  repository  for  different  sets  of  constraints  that  may  affect  parameters  of 
different  views  that  may  impact  the  human  system.  If  a  system  requires  a  human 
interface,  then  the  system  must  be  designed  to  accommodate  the  human  in  such  a  way  as 
to  account  for  the  human  limitations,  and  to  support/maintain  the  human  to  at  least  a 
minimum  acceptable  level. 

Due  to  the  range  of  information  captured  in  the  HV-B,  six  sub-products  capturing 
specific  subsets  of  data  have  been  defined  for  the  HV-B.  These  are  broken  into  two 
subsets,  Personnel,  containing  four  sub  products,  and  Human  Factors,  containing  two  sub 
products. 

Personnel  Sub  Products:  information  about  personnel  available  to  participate  in  the 
roles: 

•  Manpower  Projections  (HV-B1) — illustrates  predicted  manpower 

requirements  for  supporting  present  and  future  projects  that  contribute  to 
larger  capabilities 

o  Understand  manpower  forecasting  to  allow  initial  adjustments  in 
training,  recruiting,  professional  development,  assignment  and 
personnel  management 


78 


o  Anticipate  impacts  (and  timeframe)  related  to  number(s)  of 
personnel,  personnel  mix,  Military  Occupational  Structure 
Identification  (MOSIDs),  Rank/level  distribution,  and, 
postings/relocation(s)  of  personnel 

o  Ensure  sufficient  number  of  personnel  with  necessary  Knowledge, 
Skills,  Abilities  (KSAs)  are  ‘ready  and  able’  to  support  fielding  of 
future  program 

•  Career  Progression  (HV-B2) — illustrates  career  progression  as  well  as  the 
essential  tasks,  skills,  and  knowledge  (and  proficiency  level)  required  for  a 
given  job 

o  Address  impacts  of  alternative  system  and  capability  designs  on 
career  progression; 

o  Determine  jobs  available  given  an  individual’s  current  job  and 
occupation; 

o  Assess  competencies  required  for  each  individual  job;  and 

o  Support  personnel  planning  by  identifying  availability  of 
individuals  with  necessary  competencies  early  in  acquisition 
process 

•  Establishment  Inventory  (HV-B3) — Defines  current  number  of  personnel 
by  rank  and  job  within  each  establishment 

o  Supports  forecasting  of  trained  effective  strength 

o  Supports  predicting  number  of  people  that  must  be  trained, 
recruited,  etc.  to  fill  gaps  required  for  “out  years” 

•  Personnel  Policy  (HV-B4) 

o  Defines  the  various  department  policies  dealing  with  (governing) 
HR  issues 
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o  Ensures  that  personnel  are  fairly  considered,  properly  treated,  well 
looked  after  and  supported  in  a  legal,  moral  and  ethical  manner 
while  employed  with  the  department 

o  HR  documents,  such  as  policies,  doctrine,  laws,  benefits,  pay, 
SOPs,  etc. 

Human  Factors  Sub  Products — data  related  to  the  capabilities  of  the  humans 
assigned  to  roles: 

•  Health  Hazards  (HV-B5) 

o  Considers  the  design  features  and  operating  characteristics  of  a 
system  that  can  create  significant  risks  of  illness,  injury  or  death 

o  Aims  to  eliminate  minimize  or  control  both  short-  and  long-term 
hazards  to  health  that  occur  as  a  result  of  system  operation, 
maintenance  and  support 

o  Hazards  may  include  system,  environmental  or  task  hazard 
assessment;  air  quality  control  assessment;  noise/vibration 
pollution  evaluation;  impact  force,  shock  protection;  WHIMS 
evaluation  of  tasks;  radiation/LASER  protection;  CB  protection; 
extremes  of  temperature,  etc. 

o  It  may  include  aspects  of  survivability,  i.e.,  limiting  the  probability 
of  personal  injury,  disability  or  death  of  personnel  in  their 
interactions  with  the  system.  This  can  include  providing  protection 
from  attack,  and  reducing  detectability,  fratricide,  system  damage, 
personnel  injury  and  cognitive  and  physical  fatigue 

•  Human  Characteristics  (HV-B6) 

o  Considers  the  physical  characteristics  of  an  operator  and 
movement  capabilities  and  limitations  of  that  operator  under 
various  operating  conditions 
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o  Aims  to  compare  operator  capabilities  and  limitations  with  system 
operating  requirements  under  various  conditions  to  match  or 
eliminate  operating  capabilities 

o  It  may  include  aspects  such  as  anthropometrical/medical  data; 
reach  data;  range  of  motion  data;  physical  strength  data;  visual  and 
auditory  assessment;  speed  or  duration  of  activity  data;  cognitive 
workload;  working  memory  capacity;  ability  to  be  security  cleared; 
personality,  motivation,  etc. 

HV-C  TASKS 

The  HV-C  describes  the  human-specific  activities,  i.e.,  the  tasks  that  have  been 
assigned  to  the  humans  in  a  system  over  its  entire  life  cycle.  It  also  considers  how  the 
functions  are  decomposed  into  tasks.  (The  tenn  task  in  this  product  refers  to  a  piece  of 
work  that  can  be  assigned  to  a  person.) 

The  HV-C  may  also: 

•  Clarify  the  human-related  functions  in  a  system 

•  Provide  a  justification  for  the  allocation  of  functions  between  the  humans 
and  machines 

•  Decompose  these  functions  into  a  set  of  tasks  that  can  be  mapped  to  the 
roles  identified  in  HV-D 

•  Describe  these  tasks  in  tenns  of  various  criteria  and  the  KSA  requirements 

•  Produce  a  task-role  assignment  matrix 

•  Depict  the  inter-dependencies  between  different  tasks,  particularly  across 
functional  groupings 

•  The  information  demands  to  perform  specific  tasks 

•  The  tools  required  to  accomplish  a  task 

•  Create  interface  design  guideline  on  the  basis  of  task  requirements 
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The  HV-C  is  very  broad  and  can  be  used  to  capture  all  aspects  of  the  human- 
related  tasks  in  a  system,  including  the  allocation  of  tasks  between  humans  and  systems. 
This  product  is  also  closely  related  to  the  HV-D  Roles,  which  will  be  described  in  the 
next  section.  There  may  be  some  overlap  between  the  definition  of  tasks,  roles  and  the 
assignment  between  them.  More  often,  there  may  be  multiple  HV-C  products 
representing  different  aspects  of  the  human  tasks  in  the  architecture.  In  this  case,  the 
multiple  products  can  be  labeled  consecutively  within  the  HV-C  context. 

HV-D  ROLES 

The  HV-D  describes  the  roles  that  have  been  defined  for  the  humans  interacting 
with  the  system.  A  role  represents  a  job  function  defining  specific  behavior  within  the 
context  of  an  organization,  with  some  associated  semantics  regarding  the  authority  and 
responsibility  conferred  to  the  user  in  the  role,  and  competencies  required  to  do  the  job. 
The  role  structure  can  be  mapped  to  the  human  task  decomposition  to  define  the 
organizational  responsibilities,  and  relationships  between  roles  can  be  defined  which 
provides  the  basis  of  the  organizational  structure. 

The  HV-D  may  define  additional  attributes  of  a  role  including: 

•  Responsibility — a  form  of  accountability  and  commitment;  roles  are 
generally  defined  by  their  responsibilities 

•  Authority — is  the  access  ability  of  an  individual  user  to  perfonn  a  specific 
task 

•  Competencies — the  quality  of  being  able  to  perfonn;  a  combination  of 
knowledge,  skills  and  attributes;  these  should  be  trainable  and  measurable 

•  Multiplicity — a  role  may  be  perfonned  by  a  human  user  or  by  many 
human  users  at  the  same  time 

The  HV-D  is  closely  related  to  the  HV-C,  as  the  identified  tasks  need  to  be 
allocated  to  roles,  and  competencies  for  the  roles  defined  based  on  the  assigned  tasks. 
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The  HV-D  can  also  be  extended  to  include  a  “concept  of  work”  to  describe  the 
distribution  of  responsibilities  among  humans  and  specific  requirements  for  those 
responsibilities. 

HV-E  HUMAN  NETWORK 

The  HV-E  captures  the  human  to  human  communication  patterns  that  occur  as  a 
result  of  ad  hoc  or  deliberate  team  formation,  especially  teams  distributed  across  space 
and  time. 

Elements  of  the  HV-E  may  include: 

•  Role  groupings  or  teams  formed,  including  the  physical  proximity  of  the 
roles  and  virtual  roles  included  for  specific  team  tasks 

•  Type  of  interaction — i.e.,  collaborate,  coordinate,  supervise,  etc. 

•  Team  cohesiveness  indicators — i.e.,  trust,  sharing,  etc. 

•  Team  performance  impacts — i.e.,  synchronization  (battle  rhythm),  level  of 
engagement  (command  directed) 

•  Team  dependencies — i.e.,  frequency/degree  of  interaction  between  roles 

•  Communication/Technology  impact  to  the  team  network  -  i.e.,  distributed 
cognition,  shared  awareness,  common  operational  picture,  etc. 

HV-F  TRAINING 

HV-F  is  a  detailed  accounting  of  how  training  requirements,  strategy,  and 
implementation  will  impact  the  human.  It  illustrates  the  instruction  or  education  and  on- 
the-job  or  unit  training  required  to  provide  personnel  their  essential  tasks,  skills,  and 
knowledge  to  meet  the  job  requirements.  This  view  can  also  address  the  development  of 
additional  training  programs  to  meet  the  requirements  of  new  capabilities. 

Data  elements  of  the  HV-F  may  include: 

•  As-is  training  resources,  availability,  and  suitability 
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•  Risk  imposed  by  to-be  operational  and  system  demands 

•  Cost  and  maturity  of  training  options  for  tradeoff  analysis 

•  Address  impacts  of  alternative  system  and  capability  designs  on  training 
requirements  and  curriculums 

•  Determine  training  required  to  obtain  necessary  knowledge,  skills,  and 
ability  to  support  career  progression 

•  Differentiation  of  basic,  intermediate,  or  advance  job  training;  operational 
vs.  system  specific  training;  and  individual  vs.  team  training 

HV-G  METRICS 

The  HV-G  can  be  its  own  product  or  incorporated  into  another  architecture  metric 
view,  such  as  the  SV-7  (in  MoDAF  and  DoDAF).  It  provides  a  repository  for  human- 
related  values,  priorities  and  performance  criteria,  and  maps  human  factors  metrics  to  any 
other  human  view  elements.  It  may  map  high-level  (qualitative)  values  to  quantifiable 
performance  metrics  and  assessment  targets  or  it  may  map  measurable  metrics  to  human 
functions,  i.e.,  human  performance  specifications.  It  provides  the  basis  for  any  human 
factors  assessments  to  underpin  enterprise  performance  assessments  and  the  foundation 
for  requirements  tracking  and  certification.  For  example,  it  may  include  task  standards  as 
well  as  performance  measures. 

Elements  of  HV-G  may  include: 

•  Human  Factors  Value  definitions  level  1 . .  .n 

•  Human  Performance  Metrics  (what  is  to  be  measured) 

•  Target  Values  (what  quantifiable  value  is  acceptable) 

•  Key  Performance  Parameters 

•  Human  Task  to  Metrics  mapping 

•  Value  to  design  element  mapping 

•  Methods  of  compliance 
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HV-H  HUMAN  DYNAMICS 


The  HV-H  captures  dynamic  aspects  of  human  system  components  defined  in 
other  views.  These  are  dynamic  aspects  in  the  sense  that  states,  conditions,  or 
performance  parameters  may  change  over  time,  or  as  a  result  of  triggering  events.  It  pulls 
together  definitions  from  across  the  Human  View  to  be  able  to  communicate  enterprise 
behavior.  It  provides  inputs  to  human  behavior  and  executable  models  that  may  be 
supported  by  simulation  tools.  There  are  many  different  human  models  and  simulations 
that  can  be  used  to  develop  dynamic  models;  this  view  can  provide  stimuli  and  design 
aspects  for  these  models. 

Features  of  the  HV-H  may  include: 

•  States  (e.g.  snapshots)  and  State  Changes,  e.g., 

o  Organizational/team  structure 
o  Task/Role  assignments  to  people 
o  Team  interaction  modes 

o  Demands  on  collaboration  load  (e.g.,  need  to  spend  effort  in 
building  shared  awareness,  consensus-finding,  communicating) 

o  Task  switches/interruptions 

•  Conditions  (e.g.  triggering  events  or  situations;  scenarios) 

o  Critical  /  frequent  /  representative  /  typical  scenarios 
o  Operational  constraints  (e.g.  extensive  heat  stress) 
o  Time  conditions:  sequence,  duration,  concurrency 

•  Performance  outcomes  (observed  or  predicted),  e.g., 

o  Workload 
o  Decision  speed 
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o  Team  interaction/collaboration  style 
o  Trust  in  commanders  intent 

o  Quality  of  shared  awareness/coordination/implicit  communication 
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APPENDIX  F.  HUMAN  READINESS  LEVEL  FRAMEWORK 


The  following  pages  contain  the  first  iteration  of  the  Human  Readiness  Level 
framework.  The  contents  of  the  framework  drew  heavily  upon  the  technical  details  of  the 
NATO  Human  View  Handbook  and  the  Navy’s  HSI  Integrated  Framework  Version  1.3 
(NATO  RTO  HFM-155  Human  View  Workshop,  n.d.;  Risser,  Belk,  Smillie,  Gepp, 
2009).  At  this  point  in  the  HRL’s  evolution,  only  the  first  three  columns  have  been  fully 
populated.  The  remaining  columns  represent  the  proposed  categories  of  future  HRL 
developmental  efforts. 
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HRL 

Activity 

Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL1  = 

Activation  of 
Human  Systems 
Integration 
Process: 
Baselining  & 
Commitment 

HRL-1  activities  should 

occur  as  soon  as 
possible  prior  to 
Milestone  A  (MS  A) 

1.1 

Program  HSI  commitment 
actions;  setting  up  HSI 
infrastructure  within  SE 
planning 

1.1.1  Appoint  HSI  Lead, 

1.1.2  Initiate  Program  HSI  Plan  (HSIP)  draft 
including  HSI  Risk  mitigation  process 

1.1.3  Define  process  for  continuously  rolling  up 

HSIP  milestones  into  SEMP  at  proper  level  of 
resolution 

1.1.4  Synchronize  with  SMART,  and 

1.1.5  Establish  relationship/MOA  with  primary 
service  HSI  support  unit 

1.2 

Total  system  analysis  from  both 
functional  relationship  and 
organizational  perspectives 
with  an  empahsis  on  system 
boundaries 

1.2.1  Whole  system  Front  End  Analysis  (System, 
System  of  Systems  (SOS)and  Family  of 

Systems(FOS)  Definition); 

1.2.2  Mission  Scope  Analysis 

1.3 

HSI  Assessment  for  HRL1  (HSIA 
1):  Assessment  based  on  best 
available  solution  description 
(in  CBA,  AoA,  CONOPs,  ICD, 
etc.)  across  all  HSI  domains  to 
define  level  of  risk  in  each. 

First  priority  goal  is  to  affect 
design;  second  priority  is  to 
inform  sustainment 

1.3.1  Manpower  /Workload/  Functional  Allocation 

1.3.2  Personnel  /  KSA  clusters  /  Selection/ 

Retention/  Career  Path 

1.3.3  Training/KSAs 

1.3.4  Environment 

1.3.5  Survivability  /  Situation  Awareness/  Force 
Protection/  Egress  systems 

1.3.6  Habitabiltiy  /  Sustained  Ops/  Facilities 

1.3.7  Occupational  Health 

1.3.8  Safety/  HFACs 

1.3.9  Human  Factors  Engineering 

1.3.10  Establish  initial  issues  log  with  mitigation 
plans 

1.3.11  Discuss  and  define  HSI  risk  matrix  reporting 
process  and  criteria  for  PM  review  and  approval 

Provides  baseline  for  all 
HSI-related  activities.  All 
domains  are 
addressed.// 

Each  domain  dictates 
critical  items  of 
consideration  bearing  on 
system  design 
depending  on  the 
(initially  likely,  and  later 
definite)  Human  Touch 
Points  (HTPs)  in  a 
system  as  it  evolves. 
Risk,  for  purposes  of 
HRLs,  is  mitigated  by 
Professional  analytical 
and  managerial  activities 
that  link  critical  items  of 
consideration  to  domain- 
specific  design 
decisions,  including  well 
informed  trade-offs. 
Cricical  initial  step  in 

HSI  planning. 
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1.4 


Use  this  as  planning  tool  for  the 
HSIWG  to  ensure  maximum 
coordination  among  data-  1.4.1 
collection  and  analysis 
activities 


Integrated  activity  worksheet  for  HRL1 


1.5 


Systematic  prediction  of  errors 
that  could  be  made  by  the 
humans  in  the  system 


1.5.1  Human  Reliability  Assessment  (HRA) 


1.6 


Initial  HFE  analysis  of  projected 
system  functions,  identifying 
expected  reliability  based  on 
human  performance  vs. 
automation.  Extract  lessons 
learned  to  feed  back  to 
capability  requirements  and 
feed  forward  to  design 
implications 


1.6.1  Identify  HFE  high  drivers  and  lessons  learned 
from  legacy  systems 

1.6.2  Identify  user  needs  and  environmental 
constraints  on  human  performance 

1.6.3  Contribute  to  assessment  and  development  of 
system  concepts/  materiel  solutions 

1.6.4  Assist  in  development  of  the  cost  estimate 
based  on  anticipated  Manpower 

1.6.5  Conduct  Mission  Task  Analysis 

1. 6.5.1  Develop  Mission  Scenarios 

1. 6.5.2  Conduct  Mission  Level  Functional  Analysis 

1. 6.5.3  Develop  Design  Reference  Scenarios 

1. 6.5.4  Prelimary  Function  Allocation. 

1. 6.5.5  Conduct  Mission  Level  Task  Analysis 


Worksheets  should  be 
used  prior  to  detailing  in 
HSIP.  Several  of  the 
data  collection  activities 
are  listed  as  rows  below 
but  others  willl  have  to 
be  "planned"  based  on 
specific  proram  needs. 

A  critical  goal  should  be 
to  minimize  interference 
with  operational  SMEs 
(minimize  number  of 
visits  by  different  data 
collectiion  teams  when 
overlapping  data  are 
sought). 
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1.7 

Systematic  prediction  of 
conditions  inherent  in  normal 
operations  (including  combat) 
that  might  lead  to  death, 
disability,  illness,  injury,  or 
performance  deterioration. 
Should  Include  a  system  safety 
engineering  analysis  file  to 
document  system  design 
mitigations  as  hazards  are 
identified 

1.7.1  Hazard  Analysis/  preliminary  system,  safety 
analyses  /  Identify  safety  issues,  lessons  learned  and 
drivers  from  legacy  systems 

1.7.2  Contribute  to  assessment  and  development  of 
system  concepts  and  materiel  solutions 

1.7.3  Develop  Preliminary  Hazard  List  (PHL)  for 
each  system  concept  and  materiel  solution 

1.7.4  Assist  in  development  of  cost  estimates  based 
on  anticipated  safety  activities  &  contributions  for 
proposed  concepts/solutions 

1.7.5  Participate  in  Mission  /Task  Analysis 
[Contribute  to  TDRA  while  identifying  drivers  and 
requirements] 

1.7.6  If  munition,  perform  Threat  Hazard  Assessment 
(THA): 

1.7.7  Generate  PHL 

1.8 

Identify  manning  needs, 
manpower  lessons  learned, 
and  high  drivers  from  legacy 
syste  ms 

1.8.1  Establish  manpowr  thresholds  and  objectives 

1.8.2  Establish  level  and  duration  criterion  for  high 
sustained  workload. 

1.8.3  Identify  manpower  goals  and  parameters 
(DoD1 100.4) 

1.8.4  Define  the  human  performance  characteristics 
of  the  user  population  (based  on  mission  and  system 
requirements,  target  occupational  specialties, 
recruitment  and  retention  trends) 

1.8.5  Contribute  to  assessment  and  development  of 
system  concepts  /  materiel  solutions 

1.8.6  Assist  in  development  of  the  cost  estimate 
based  on  anticipated  manpower 

1.8.7  Participate  in  Mission  Task  Analysis 
[Incorporate  findings  from  Mission  Task  Analysis  into 
initial  TDRA) 
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1.9 


Synchronize  with  DoDAF 


1.9.1  Coordinate  with  Requirements  Manager  (RM) 
on  current  state  of  OV-1  and  CONOPS; 

1.9.2  Begin  specification  of  HV-A 

1. 9.2.1  Review  information  and  data  components 
required  to  begin  populating  all  HVs 


1.10' 


Linear  modeling  and  studies 
/analyses  of  System  Approach 
Alternatives 


1.10.1  Plan  preliminary  linear  models 

1.10.2  Negotiate  with  stakeholders  of  "what  if' 
studies 

1.10.3  Begin  coordination  with  Man  In  the  Loop 
(MIL)  simulation  experiments 


1.11a 


Perform  personnel  alternatives 
concept  development  and 
assessments 


1.11.1  Consider  personnel  alternatives  based  on 
FSA  (CJCSM  /3170.01C) 

1.11.1.2  Determine  impacts  on  cost  estimates 

1.11.1.3  Determine  appropriate  personnel 
approaches 

1.11.1.4  Assist  in  development  of  the  cost  estimate 
based  on  anticipated  personnel  activities 

1.11.2  Define  the  human  performance 
characteristics  of  the  user  population 
(based  on  mission  and  system  requirements, 
target  occupational  specialties,  and 
recruitment  and  retention  trends) 


1.11b 


Perform  personnel  alternatives 
concept  development  and 
assessments 


1.11.3  Identify  personnel  shortfalls 

1.11.4  Use  findings  from  the  mission  task  analysis 
and  other  personnel  assessments  to  identify  KSAs 

1.11.5  Assess  personnel  readiness,  personnel 
tempo,  and  funding  issues  that  may  impact  the 
ability  to  fill  personnel  requirements 

1.11.6  Perform  an  analysis  of  alternatives  for 
personnel  optimization 


OV-1:  operational 
concept  graphic. 
Concept  (HV-A) 
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1.12.1  Perform  a  training  effectiveness  evaluation 
on  legacy  systems 

1.12.1.1  Provide  recommendations  to  correct 
training  deficiencies 

1.12.1.2  Identify  training  deficiencies  and  root 


Perform  training  alternatives 
concept  development  and 


1.12.2  Assess  human  performance  and  effectiveness 
of  training  transfer 

1.12.3  Assist  in  development  of  the  cost  estimate 
based  on  anticipated  training  activities 

1.12.4  Participate  in  Mission  Task  Analysis 

1.12.4.1  Input  mission  scenarios  to  the  training 
assessment 

1.12.4.2  Conduct  assessments  to  determine  KSAs, 
learning  objectives,  and  training  requirements  for 
the  proposed  materiel  solution 

1.12.4.2.1  Justify  the  training  system 

1.12.4.2.2  Skills  and  Training  analysis 

1.12.4.2.3  Training  Requirements  Analysis 

1.12.4.2.4  Complete  total  training  requirements 

1.12.4.2.5  Perform  a  media  selection  analysis 


Perform  training  alternatives 
concept  development  and 


1.12.5  Perform  analysis  of  alternatives  for  training 
optimization 

1.12.6  Technologies/cost  trade-off 
analyses  for  training  options 

1.12.6.1  Assist  in  development  and  evaluation  of  the 
training  program  planning  of  alternatives 

1.12.6.2  Consider  tasks,  conditions,  and  standards 
from  the  Functional  Area  Analysis 
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HRL 

Activity 

Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL2=HSI 

analysis  Suite  to 
support 
component 
technology 
development 

HRL-2  activities 

should  occur  as 
soon  as  possible 
prior  to  MS  A 

2.1 

HSI  Assessment  -  HRL2 
(HSIA2).  First  priority  goal  is  to 
affect  design;  second  priority  is 
to  inform  sustainment. 

2.1  Assess  HSI  considerations  in  each  Domain 

2.1.1  Manpower  /Workload/  Functional  Allocation 

2.1.2  Personnel  /  KSA  clusters  /  Selection/ 

Retention/  Career  Path 

2.1.3  Training /KSAs 

2.1.4  Environment/ 

2.1. .5  Survivability  /  Situation  Awareness/  Kll  Chain 
Analysis/  Force  Protection/  Egress  systems 

2.1.6  Habitabiltiy  /  Sustained  Ops/ Facilities 

2.1.7  Occupational  Health 

2.1.8  Safety/  HFACs 

2.1.9  Human  Factors  Engineering 

2.1.9.1  Define  HFE  Concept/Capability 

2.1.10  HSI  WG  review  HSI  Issues  log  and  mitigaton 
progress 

2.1.11  HSI  WG  specifies  and  recommends  risk 
matrix  outcomes  and  HRL  assignment  to  PM 

Re-Assessment  based 
on  best  available 

Solution  Description 
(Now  to  include  solution 
alternative  from  AoA  or 
Block  definition  and 
current  state  of 
CONOPs,  ICD,  etc.) 
across  all  HSI  domains 
to  define  level  of  risk  in 
each.  Each  domain 
dictates  critical  items  of 
consideration  bearing  on 
system  design 
depending  on  the 
(initially  likely,  and  later 
definite)  HTPs  in  a 
system  as  it  evolves. 
Risk,  for  purposes  of 
HRLs,  is  mitigated  by 
Professional  analytical 
and  managerial  activities 
that  link  critical  items  of 

consideration  to  domain- 
specific  design 
decisions,  including  well 
informed  trade-offs. 
Provides  baseline  for  all 
HSI-related  activities.  All 

Domains  are  addressed. 

93 


2.2 


Update/revise  HSIP 


2.2.1  Using  worksheet  as  described  in  HRL-1  to 
update  sequence  of  activities  remaining 


2.3 

Generate  Target  Audience 
Description  and  Airman 
Capabiltiy  Document 

2.3.1  Use  findings  from  the  mission  task  analysis 
and  other  personnel  assessments  to  identify  KSAs  / 
Human  Performance  requirements 

2.3.2  Analysis  activities  to  define  Manpower, 
Personnel  and  Training-related  Human  Performance 
parameters 

2.3.3  Assess  personnel  /  readiness,  personnel 
/tempo,  and  funding  issues  that  may  impact  the 
ability  to  fill  Personnel  requirements/current  and 
needed  AFSCs  /  hard  to  fill  occupations  /  required 
KSAs 

2.3.4  Derive  initial  Personnel  requirements/[New 
occupational  specialties  (unique  skill  sets)  /hard  to 
fill  occupations  Highly  qualified  personnel. 

Determine  how  requirements  will  be  met 

2.4a 

Training  Systems  Requirements 
Analysis  (TSRA) — The  initial 
step  in  user  requirements 
identification.  It  consists  of 
mission/task  analysis,  training 
requirements  identification, 
objectives/media  analysis,  and 
training  systems  basis  analysis. 
A  TSRA  integrates  the  products 
of  the  Instructional  System 
Development  (ISD)  process  and 
the  Systems  Engineering  (SE) 
process  to  describe  the 

Training  System  to  be 
procured.  It  serves  as  a 
required  input  to  the  System 
Training  Plan.  It  is 
accomplished  by  the  PM  in 
coordination  with  the  Lead 
Command  and  User  Command 
(Aug  30,  2007  ...  This  TSRA 
process  is  consistent  with  DODD 
1430.13) 

2.4.1  Participate  in  Mission  Task  Analysis 

2.4.2  Input  mission  scenarios  to  the  training 
assessment//JTA  reviews 

2.4.3  Conduct  assessments  to  determine  KSAs, 
learning  objectives,  and  /training  requirements  for 
the  proposed  materiel  solution 

2.4.3. 1  Justify  the  training  system  Learning 
Performance  and  Solution  Analysis 

2.4.3.2  Skills  and  Training  Analysis/Training 
Requirements  Analysis 

2. 4.3.3  Complete  total  training  requirements 

2.4.4  Perform  a  high  level  media  selection  analysis 

2. 4.5.1  Perform  an  analysis  of  alternatives  for 
training  optimization 

2.4.5.2  New  learning  Technologies  Cost  trade-off 
analyses  for  training  options 

2. 4.5.3  Assist  in  development,  and  evaluation  of  the 
training  /  program  planning 

2. 4.5.4  Provide  descriptions,  methods,  and 
technology  requirements  for  the  training  concept 
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2.4b 


Training  Systems  Requirements 
Analysis  (TSRA)— The  initial 
step  in  user  requirements 
identification.  It  consists  of 
mission/task  analysis,  training 
requirements  identification, 
objectives/media  analysis,  and 
training  systems  basis  analysis. 
A  TSRA  integrates  the  products 
of  the  Instructional 
SystemDevelopment  (ISD) 
process  and  the  Systems 
Engineering  (SE)  process  to 
describe  the  Training  System  to 
be  procured.  It  serves  as  a 
required  input  to  the  System 
Training  Plan.  It  is 
accomplished  by  the  PM  in 
coordination  with  the  Lead 
Command  and  User  Command 
(Aug  30,  2007...  This  TSRA 
process  is  consistent  with  DODD 
1430.13) 


2.4.5.5  Consider  tasks,  conditions,  and  standards 
from  the  Functional  Area  Analysis  (CJCSM  /3170.01C) 

2.4.6  Educational  analysis.  A  process  of  reviewing 
the  educational  requirements,  developing 
educational  goals,  and  developing  statements  for 
achieving  the  goals. 

2.4.7  Occupational  analysis.  A  process  of  identifying 
tasks  that  are  closely  related  and  grouped  under  an 
occupational  specialty. 

2.4.8  Mission  analysis.  A  process  of  reviewing 
mission  requirements,  and  developing  a  mission 
statement,  mission  segments,  collective  task 
statements,  and  arranging  the  segments  and  tasks  in 
a  hierarchical  relationship. 

2.4.9  Job  analysis.  Identifies  the  jobs  that  define  an 
occupational  skill  area  and  identifies  duties  and 
tasks  that  comprise  each  job 


2.5 


2.5.1  Complements  2.3,  specialized  HFE  interview 
and  validation  process  that  translates  operator/ 
Legacy  Operator/Maintainer  maintainer  interpretations  of  mission  activities  into 
Vignettes  process  sequence  descriptions  that  systems 

engineers  can  use  as  a  basis  for  system  specification 
and  design 


2.6 


Generate  Design  Reference 
Missions 


2.6.1,  Enabled  by  2.5,  Mission  descriptions  that  in 
combination  tap  the  full  range  of  functional 
capabilities  of  the  system 


2.7 


Perform  Function  Allocation 
analysis 


2.7.1  Much  more  informed  version  of  work  begun  in 
1.6,  allocating  tasks  to  humans  and  machines  in  a 
way  that  adds  value  to  capability  in  term  of  cost, 
safety  and  performance.  Exploits  relative  strengths 
of  man  vs.  machine!! 


2.8 


Human  Reliability  Assessment 
(HRA) 


2.8.1  Based  on  predictions  about  com ponte nt 
technologies,  systematic  prediction  of  errors  that 
could  be  made  by  the  humans  in  the  system 


2.9 


Update  of  Hazard  Analysis, 
Threat/Hazard  Assessment 
(THA):  Preliminary  Hazard 
Analysis  (PHA)  and 
Requirements  Hazard  Analysis 
(RHA) 


2.9.1  Systematic  prediction  of  conditions  inherent  in 
normal  operations  (including  combat)  that  might 
lead  to  death,  disability,  illness,  injury,  or 
performance  deterioration. 

2.9.2  Derive  initial  SOH  requirements/ 

2.9.3  Update  hazaards  lists 


2.10' 


Initial  Manpower  Estimate 
Analysis;  Generate  TDRA  (Top 
Down  Requirements  Analysis) 


2.10.1  Based  on  analyses  to  date,  updated  modeling 
and  analysis  against  operational  needs  to  generate 
manpower  estimate  working  with  A1  tools, 
representatives  and  process  stakeholders  (LCOM, 
etc.) 

2.10.2  Conduct  initial  TDRA  [Compute  time-on-task 
for  readiness  conditions  and  mission  scenarios 

2.10.3  Produce  data  to  support  manpower 
determination  from  the  allocated  functions  and  tasks 
/  Initial  Workload  Analysis 

2.10.4  Perform  an  analysis  of  alternatives  for 
manpower  optimization  [Identify  capability  gaps  in 
manpower  reduction.  Quantify  workload  reductions 
for  each  specific  alternative  (AoA)] 

2.10.5  Derive  initial  Manpower  requirements 
/Identify  constraints  and  limitations 
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2.11 

Ergonomics/biomechanical  / 
anthropometric  assessment 

2.11.1  Preliminary  analysis  of  notional  human  touch 
point  (HTP)  taking  into  account  form,  fit  and  function 
to  accommodate  both  static  and  dynamic  human  at 
work;  includes  accessability  and  egress  during 
emergency  situations  and  normal  work 

2.12 

Initial  Human  Machine 

Interface  (HMI)  assessment 

2.12.1  Preliminary  analysis  of  interface  design 
considerations  primarily  related  to  controls  and 
displays  but  extending  to  any  human  touch  point, 
with  or  without  the  use  of  tools  and  the  design  and 
diversity  of  the  tools  themselves 

2.13 

Review  LCMP  draft 

2.13.1  Identify  stakeholders  and  update  assumptions 
for  current  program  state 

2.14 

Review  TEMP  draft 

2.14.1  Identify  stakeholders  and  update  assumptions 
for  current  program  state 

2.15 

Activate  risk  management 
process 

2.15.1  Establish  detailed  administrative  processes 
for  project  HSI  issues  log  (in  HSIP) 

2.16 

Syncronize  with  DoDAF  and  or 
incorporate/roll-up  into  SEMP 

2.16.1  Coordinate  with  RM  on  current  state  of  OV4 
and  CONOPS;  begin  specification  of  HV-B,  D-1,  and 
F-1.  Review  information  and  data  components 
required  to  begin  populating  all  HVs 

2.17 

Consider  high  ROI  use  of  mock- 
ups  for  physical  layouts  of  HTPs 

2.17.1  Formally  consider  /  plan  generating  mock- 
ups  to  allow  high-level  assessment  of  safety, 
communications,  workflow,  usability,  and  general 
form  and  fit  of  design  alternatives.  These  are 
especially  useful  when  experienced  legacy  system 
operators  and  maintainersare  involved  in  these 
assessments 

2.18 

Incorporate  requirement  for 
contractor  generated  HSIP(s) 
and  associated  source 
selection  criteria  in  Request  for 
Proposals  (RFPs) 

2.18.1  Review  RFP  language,  CDRLsand  DIDsfor 

HSi  Planning  activities  for  each  Human 

Perforrmance 

2.19 

Apply  analysis  findings  to  Test 
and  Evaluation  Strategy  (TES) 
and  TEMP 

2.19.1  Special  analysis  to  update  thresholds, 
objectives,  and  evolving  criteria  for  DT&E  and 
eventually  OT&E 

2.20' 

Conduct  formal  development  of 
human-centered  source 

selection  criteria  for  RFPs  at 
Milestone  (MS)  A 

2.20.1  HSI  review  of  Source  Selection  Criteria  for 
developmental  RFPs 
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HRL 

Activity 

Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL3= 

Component 
Human  Touch 
Point  Resolution: 
Refining 
Requirements 
Thresholds 

HRL -3  activities 

should  occur  as 
soon  as  possible 
after  MS  A,  prior  to 
MS  B.  Nominally 
about  mid-way  to 

3.1 

HSI  Assessment  -  HRL3  (HSIA- 
3).With  formal  conversion  from 
portfolio/project  to  Program  at 
MSA,  PM  must  assess 
completion  of  HRL  1  &2 
activities  with  emphasis  on 
continuity  and  partnership  with 
MAJCOM  RM  HSI  WG/IPT  MSA 
Phase  analytic  activities  to  date 

3.1.1  Manpower /Workload/  Functional  Allocation 

3.1.2  Personnel  /  KSA  clusters  /  selection/  retention/ 
career  path 

3.1.3  Training  / KSAs 

3.1.4  Environment 

3.1.5  Survivability  /  Situation  Awareness/  Kll  Chain 
Analysis/  Force  Protection/  Egress  systems 

3.1.6  Habitabiltiy  /  Sustained  Ops/ Facilities 

3.1.7  Occupational  Health 

3.1.8  Safety/ HFACs 

3.1.9  Human  Factors  Engineering 

3.1. 9.1  Refine  user  needs,  characteristics,  and 
environmental  constraints 

3.1. 9.2  Support  prototyping  and  capability  based 
technology  down-selection 

3.1.10  HSI  WG  review  HSI  issues  log  and  mitigaton 
progress 

3.1.11  HSI  WG  specifies  and  recommends  risk 
matrix  outcomes  and  HRL  assignment  to  PM 

3.2 

Update/revise  HSIP 

3.2.1  Using  worksheet  as  described  in  HRL-1  to 
update  sequence  of  activities  remaining 
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3.3 

Analysis  to  support  manpower 
estimates 

3.3.1  Emphasis  on  updating  manning  estimates  as 
component  HTPs  are  resolved 

3.3.2  Update  ME  data  where  appropriate 

3.4 

Second  iteration  of  TSRA 

3.4.1  Update  ISD  analyses 

3.4.2  Explore  training  technology  options  in 
components  (embedded  training?;  distributed 
training?) 

3.5 

Update  HSI  issues  log 

3.5.1  Emphasis  on  HFE  design  resolutions  of  issues 

3.6 

Initial  estimates  of  MPT  costs 

3.6.1  Increasing  detail  in  cost  estimation  process 

3.6.1. 1  Participate  in  HSI  IPT:  Perform  analyses  to 
support  development  of  the  ME 

3.6.1.2  Refine  and  finalize  ME 

3.7 

Syncronize  with  DoDAF  and  or 
incorporate/roll-up  into  SEMP 

3.7.1  Provide  updated  information  to  RM  for 
continued  expansion  of  OVs  5,  6,  &  7,  and  CONOPS; 

3.7.2  Continue  specification  of  HV-C-1  and  begin 
populating  all  HVs  as  component  data  evolves. 

3.7.3  Update  design  reference  mission  scenarios  as 
needed. 

3.8 


Ensure  HSIA  updated  for  any 
newly  introduced  component 
technologies 


3.8.1  Update  HSIA  for  any  introduced  component 
technology.  (Especially  important  for  commercial  off 
the  shelf  (COTS)) 


3.9 

Initiate  Cognitive  Task  Analysis 
Process  to  inform  component 
technology  developments 

3.9.1  Begin  component  Cognitive  Task  Analyses 
(feeds  over  to  3.7.2  and  3.7.3) 

3.10. 

Update  THA  and  HRA 

3.10.1  Update  either  HRA  or  THA  or  both 

3.10.2  Udate  HFACs  or  similar  nanacodes  as 
components  gel 

3.11 

Initiate  Ergonomics/ 
biomechanical  / 
anthropometric  assessment 
processes 

3.11.1  Analysis  of  evolving  HTPs  taking  into  account 
form,  fit  and  function  to  accommodate  both  static 
and  dynamic  human  at  work;  includes  accessability 
and  egress  during  emergency  situations  and  normal 
work 

3.12 

Plan  modeling  to  inform 
human-in-the-loop  analyses 
and  assessments  based  on  cost 

benefit 

3.12.1  Conduct  analysis  and  negotiate  specification 
of  Independent  Variables  (IVs)  for  linear  modeling. 
Select  IVs  in  part  based  on  factors  that  can  be 
manipulated  in  man-in-the-loop  ((MITL)  usually 
simulation-based)  experiments.  Modeling  then  can 
reduce  complex,  multivariable  solution  space 
followed  by  iterative  MITL  validation  in  medium- to 
high-fidelity  simulation 

3.13 

Update  Human  Machine 
Interface  (HMI)  assessment 

3.13.1  Analysis  of  interface  design  considerations 
primarily  related  to  controls  and  displays  but 
extending  to  any  human  touch  point,  with  or  without 
the  use  of  tools  and  the  design  and  diversity  of  the 
tools  themselves 

3.14 

Consider  high  ROI  use  of  mock- 
ups  for  physical  layouts  of  HTPs 

3.14.1  Evaluate  use  of  mock-ups  to  allow  high-level 
assessment  of  safety,  communications,  workflow, 
usability,  and  general  form  and  fit  of  design 
alternatives.  These  are  especially  useful  when 
experienced  legacy  system  operators  and 
maintainersare  involved  in  these  assessments 
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3.15 

Subsystem  Hazard  Analysis 
(SSHA) 

3.15.1  Emerging  Subsystem  Hazard  Analysis  (SSHA) 

3.16 

Apply  analysis  findings  to  TEMP 

3.16.1  Special  analysis  to  update  thresholds, 
objectives,  and  evolving  criteria  for  DT&E  and  OT&E 

HRL 

Activity 

Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL4= 

Component 
Human  Touch 
Point 

Engineering 
Parameters  and 
Human 
Performance 

HRL-4  activities 

should  occur  as 
soon  as  possible 
after  MS  A  and  prior 
to  MS  B.  Nominally 
prior  to  SRR. 

4.1 

HSI  Assessment  -  HRL4  (HSIA- 
4) 

4.1.1  Assessment  based  on  best  available  solution 
description  across  all  HSI  domains  to  define  level  of 
risk  in  each.  Each  domain  dictates  critical  items  of 
consideration  bearing  on  system  design  depending 
on  the  (initially  likely,  and  later  definite)  HTPs  in  a 
system  as  it  evolves.  Risk,  for  purposes  of  HRLs,  is 
mitigated  by  professional  analytical  and  managerial 
activities  that  link  critical  items  of  consideration  to 
domain-specific  design  decisions,  including  well- 
informed  trade-offs. 

4.1.2  HSI  WG  review  HSI  issues  log  and  mitigaton 
progress 

4.1.3  HSI  WG  specifies  and  recommends  risk  matrix 
outcomes  and  HRL  assignment  to  RM/PM 
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4.2 


Update/revise  HSIP 


4.2.1  Using  worksheet  as  described  in  HRL-1  to 
update  sequence  of  activities  remaining 


4.3 

Update/revise  LCMP 

4.3.1  Update  current  LCMP 

4.4 

Update  Human  Machine 
Interface  (HMI)  assessment 

4.4.1  Analysis  of  interface  design  considerations 
primarily  related  to  controls  and  displays  but  must 
be  extended  to  any  human  touch  point,  with  or 
without  the  use  of  tools  and  the  design  and  diversity 
of  the  tools  themselves. 

4.4.2  Perform  technology  assessments  (with 
representative  users)  and  evaluate  HFE  based 
technology/system  requirements,  user 
characteristics,  and  human  performance 

4.4.3  Cognitive  walkthroughs/HMI/HCI  Evaluations/ 
Human  Performance  Evaluations  and  Risk  Analyses 

4.4.4  Conduct  trade  studies  to  assess  trade-offs 
between  technologies. 

4.4.5  Conduct  trade  studies  to  assess  trade-offs 
between  technologies. 

4.4.6  Identify  HFE  technical  risks  and  mitigation 
strategies  included  specification  of  applicable 
guides  and  standards 

4.4.7  Provide  technology  down  selection 
recommendations 

4.4.8  Review  sequencing  and  execution  of 
component  usability  testing 
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4.5 


Consider  high  ROI  use  of  mock- 
ups  for  physical  layouts  of  HTPs 


4.5.1  Formally  consider  mock-ups  to  allow  high- 
level  assessment  of  safety,  communications, 
workflow,  usability,  and  general  form  and  fit  of 
design  alternatives.  These  are  especially  useful 
when  experienced  legacy  system  operators  and 
maintainersare  involved  in  these  assessments 


4.6 


Analysis  of  support  availability 
for  training  needs  and 
manpower  /  personnel 
requirements 


4.6.1  Communicate  with  RM  and  HSI  functionals  to 
take  initial  opportunity  to  bring  critical  stakeholders 
from  using  and  supporting  commands  into  the  loop 
for  planning  long-term  training  and  manpower 
support  needs.  Also,  provides  critical  opportunity  for 
early  id  and  resolution  of  evolving  training  system 
gaps. 

4.6.2  Participate  in  technology  assessments  to 
evaluate  adequacy  of  manpower  estimates  and 
operational  impacts  of  proposed  manning  solutions. 

4.6.3  Refine/update  workload  for  specific 
technology  ops  and  support  activities 

4.6.4  Develop  manpower  reduction  strategy 

4.6.5  Assess  manpower  tradeoffs  among 
technologies 

4.6.6  Participate  in  technology  assessments  to 
evaluate  personnel  requirements  and  the  adequacy 
of  personnel-related  content  in  the  TSP 

4.6.7  Assess  personnel  trade-offs  between 
technologies 


4.7 


Subsystem  Hazard  Analysis 
(SSHA) 


4.7.1  Conduct/update  Subsystem  Hazard  Analysis 
(SSHA) 


Iterative  Tranining  system 
evolution 


4.8.1  Participate  in  technology  assessments  to 
evaluate  adequacy  of  the  TSP  and  operational 
impacts  of  proposed  training  solutions. 

4.8.2  Identify  and  document  formal  training  courses 
and  training  resource  requirements 

4.8.3  Type  of  training  curricula 

4.8.4  Ensure  sufficient  funding  to 
support  training  needs, training 
devices,equipment 

4.8.5  Assess  training  trade-offs  between 
technologies 


Apply  analysis  findings  to  TEMP 


4.9.1  Conduct  special  analysis  to  update  thresholds, 
objectives,  and  evolving  criteria  for  DT&E  and  OT&E 


Begin  formal  development  of 
human-centered  source 
selection  criteria  for  RFPs  at 
MS  B 


4.10.1  Based  on  requirements  being  translated  into 
RFPs,  begin  drafting  HSI-specific  source  selection 
criteria 


4.11.1  Provide  updated  information  to  RM  for 
continued  expansion  of  OVs  5,  6,  &  7,  and  CONOPS; 
Syncronize  with  DoDAF  and  or  4.11.2  Continue  specification  of  HV-C-1  and  continue 
incorporate/roll-up  into  SEMP  populating  all  HVs  as  component  data  evolves. 

4.11.3  Update  design  reference  mission  scenarios  as 
needed 
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Activity 

Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL5=Limited 
System  Human 
Performance 
Parameters/ 
Demonstration 

HRL -5  activities 

should  occur  as 
soon  as  possible 
after  MS  A  and  prior 
to  MS  B.  Nominally 
prior  to  SFR. 

5.1 

HSI  Assessment  -  HRL5 
(HSIA-5) 

5.1.1  Assessment  based  on  best  available  Solution 
Description  across  all  HSI  domains  to  define  level  of 
risk  in  each.  Each  domain  dictates  critical  items  of 
consideration  bearing  on  system  design  depending 
on  the  (initially  likely,  and  later  definite)  HTPs  in  a 
system  as  it  evolves.  Risk,  for  purposes  of  HRLs,  is 
mitigated  by  professional  analytical  and  managerial 
activities  that  link  critical  items  of  consideration  to 
domain-specific  design  decisions,  including  well- 
informed  trade-offs. 

5.1.2  HSI  WG  review  HSI  Issues  log  and  mitigaton 
progress 

5.1.3  HSI  WG  specifies  and  recommends  risk  matrix 
outcomes  and  HRL  assignment  to  RM/PM 

5.2 

Update/revise  HSIP 

5.2.1  Using  worksheet  as  described  in  HRL-1  to 
update  sequence  of  activities  remaining 
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5.3 


5.3.1  Refine/update  Personnel  requirements  /  KPP 
and  KSA  in  CDDs 

5.3.2  Assist  in  final  technology  selections/Ensure 
Personnel  needs  can  be  met 
appropriately/addresses  Personnel  requirements 


5.4 


Confirm  SOH  findings 
incorporated  in  design  and  fed 
forward  to  sustainment 
planning 


5.4.1  Confirm  that  safety  and  Occupational  Health 
design  features  have  been  incorporated 

5.4.2  Non-materiel  safety  and  health  hazard 
mitigations  have  been  incorporated  into  PESHE 


5.5 


Subsystem  Hazard  Analysis 
(SSHA) 


5.5.1  Review  component  technologies  and  update 
Hazard  Analyses 


5.6 


Update  Human 
Machine/Equipment  Interface 
(HMI)  assessment 


5.6.1  Refine/update  HFE  requirements 

5.6.2  HMI/  HCI  usability  assessments  of  component 
Workspaces/  Communication  systems/  Human 
Performance  &  Safety/Occ  Health  features/ 

5.6.3  Refine  KPPsand  KSAsin  CDD 

5.6.4  Assist  in  final  technology  selections,  assure 
solution: 

5.6.4.1  Addresses  HFE  requirements 

5.6.4.2  Addresses  user  needs  and  goals 

5.6.4.3  Satisfies  concept  and  mission  objectives 

5.6.4.4  Incorporates  HFE  design  standards 
5.6.5  Component  performance  testing.  Update 
predictive  models  based  on  outcomes 
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5.7 


Update  ME 


5.7.1  Refine/update  Manpower  requirements 

5.7.2  Assure  final  technology  selections 
addresses  Manpower  requirements 

5.7.3  Ensure  MEscan  be  met  appropriately 


5.8 


Apply  analysis  findings  to  TEMP 


5.8.1  Special  analysis  to  update  thresholds, 
objectives,  and  evolving  criteria  for  DT&E  and  OT&E 


5.9 


Translate  design  decisions  into 
design  specifications  and/or  life 
cycle  sustainment  factors 


5.9.1  Continue  formal  development  of  human- 
centered  source  selection  criteria  for  RFPs  at  MS  B 

5.9.2  Update/revise  LCMP 


5.10. 


Syncronize  with  DoDAF  and  or 
incorporate/roll-up  into  SEMP 


5.10.1  Syncronize  with  DoDAF  and  or 
incorporate/roll-up  into  SEMP 


5.11 


Update  Training  for  design 
decisions  and  update 
sustainment  accordingly 


5.11.1  Update  Training  for  design  decisions  and 
update  sustainment  accordingly 


5.12 


Synchronize  with  DoDAF 


5.12.1  Ensure  that  human  reliability  issues  have 
been  identified  for  action  in  design  or  appropriate 
process  within  the  HV 

5.12.2  Provide  updated  information  to  RM  for 
continued  expansion  of  OVs  5,  6,  &  7,  and  CONOPS; 
continue  specification  of  HV-C-1  &  D-2,  and 

5.12.3  Continue  populating  all  HVs  as  component 
data  evolves. 

5.12.4  Update  design  reference  mission  scenarios  as 
needed 
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Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL6=Field 
(Relevant 
Environment) 
Validation  of 
Human 
Performance 
Prototypes 

HRL -6  activities 
should  occur  prior  to 
MS  B. 

6.1 

HSI  Assessment- HRL6  (HSIA- 
6) 

6.1.1  Assessment  based  on  best  available  Solution 
Description  across  all  HSI  domains  to  define  level  of 
risk  in  each.  Each  domain  dictates  critical  items  of 
consideration  bearing  on  system  design  depending 
on  the  (initially  likely,  and  later  definite)  HTPs  in  a 
system  as  it  evolves.  Risk,  for  purposes  of  HRLs,  is 
mitigated  by  Professional  analytical  and  managerial 
activities  that  link  critical  items  of  consideration  to 
domain-specific  design  decisions,  including  well- 
informed  trade-offs. 

6.1.2  HSI  WG  review  HSI  Issues  log  and  mitigaton 
progress 

6.1.3  HSI  WG  specifies  and  recommends  risk  matrix 
outcomes  and  HRL  assignment  to  RM/PM 

6.2 

Update/revise  HSIP 

6.2.1  Using  worksheet  as  described  in  HRL-1  to 
update  sequence  of  activities  remaining 
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6.3 


Update/revise  LCMP 


6.3.1  Update/revise  LCMP 


6.4 

Subsystem  Hazard  Analysis 
(SSHA) 

6.4.1  Subsystem  Hazard  Analysis  (SSHA) 

6.5 

System  Hazard  Analysis  (SHA) 

6.5.1  Perform  system  safety  analyses 

6.5.2  Derive  and  finalize  safety  requirements  and 
Criteria  for  Analysis,  6.5.2.1  develop  preliminary 
hazard  list 

6.5.2.2  Initiate  preliminary  hazard  analysis  and 
threat  hazard  assessment 

6.5.3  Complete  initial  /PESHE 

6.6 

Iterative  usability  testing 

6.6.1  Conduct  Evaluations  of  Human  Perfomance 
embedded  in  demontration  system; 

6.6.2  Update  predictive  models  based  on  outcomes 

6.7a 

Apply  analysis  findings  to  TEMP 

6.7.1  Special  analysis  to  update  thresholds, 
objectives,  and  evolving  criteria  for  DT&E  and  OT&E 

6.7.2  Participate  /in  the  T&E  /IPT 

6.7.3  Perform  analyses  to  ensure  appropriate  HFE 
considerations  are  included  in  system  technology 
development//  Operational  /Workflow  Analysis/  Task 
Analyses 

6.7.4  Assist  in  development  of  the  system  functional 
baseline  and  Functional  Allocation  Analyses 

6.7.5  Develop  the  HFE  T&E//V&V  program 
plans//Methods/Metrics  /Analyses  /Tools 

6.7.6  Review  and  provide  inputs  to  preliminary 
designs  allocated  baseline 
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6.7b 


Apply  analysis  findings  to  TEMP 


6.7.7  Provide  recommendations  based  on 
Manpower  analyses 

6.7.8  Assess  gaps  between  personnel  capabilities 
and  system  needs 

6.7.9  Provide  Personnel  inputs  to  functional 
allocation 

6.7.10  Provide  recommendations  based  on 
Personnel  analyses 

6.7.11  Assess  gaps  between  personnel  capabilities 
and  system  needs 

6.7.12  Provide  Personnel  inputs  to  functional 
allocation 

6.7.13  Provide  recommendations  based  on  Training 
analyses 


6.8 


Complete  formal  development 
of  human-centered  source 
selection  criteria  for  RFPs  at 


6.8.1  Complete  formal  development  of  human- 
centered  source  selection  criteria  for  RFPs  at  MS  B 


MS  B 


6.9.1  Provide  updated  information  to  RM  for 
continued  expansion  of  OVs  2,  3,  &  5,  and  CONOPS; 
continue  specification  of  HV-E-1,  2,  &3,  HV-C-2,  and 


6.9 


Syncronize  with  DoDAF 


HV-G. 

6.9.2  Continue  populating  all  HVs  as  component 
data  evolves. 

6.9.3  Update  design  reference  mission  scenarios  as 
needed 


Collect  evidence  for  validation 
and  verification  (V&V),  and 
acceptance  against  HSI 
requirements,  including 
interoperability. 


6.10.1  Collect  evidence  for  validation  and 
verification  (V&V),  and  acceptance  against  HSI 
requirements,  including  interoperability 


6.10. 


HRL 

Activity 

Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL7=Final 
DT&E  Human 
Performance 
Parameters 

HRL-7  activities  should 

occur  as  soon  as 

possible  after  MS  B. 
Nominally  by  CDRA. 

7.1 

HSI  Assessment  -  HRL7  (HSIA- 
7) 

7.1.1  Assessment  based  on  best  available  Solution 
Description  across  all  HSI  domains  to  define  level  of 
risk  in  each.  Each  domain  dictates  critical  items  of 
consideration  bearing  on  system  design  depending 
on  the  (initially  likely,  and  later  definite)  HTPs  in  a 
system  as  it  evolves.  Risk,  for  purposes  of  HRLs,  is 
mitigated  by  Professional  analytical  and  managerial 
activities  that  link  critical  items  of  consideration  to 
domain-specific  design  decisions,  including  well- 
informed  trade-offs.  Emphasis  at  this  stage  should  be 
optimizing  transfer  of  HSI  issues  and  consideration 
information  to  the  sustainment  community,  most 
specifically  logistics  center  managers. 

7.1.2  HSI  WG  review  HSI  Issues  log  and  mitigaton 
progress 

7.1.3  HSI  WG  specifies  and  recommends  risk  matrix 
outcomes  and  HRL  assignment  to  PM 

7.2 

Update/revise  HSIP 

7.2.1  Using  worksheet  as  described  in  HRL-1  to 
update  sequence  of  activities  remaining 
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Update/revise  LCMP 


7.3.1  Assess  detailed  system  designs  to  ensure  they 
meet  Manpower  requirements 

7.3.2  Participate  in  detailed  system  design 
demonstrations 

7.3.3  Provide  recommendations  for 
redesign/modification  and/or  develop  mitigation 
strategies  for  Manpower  estimates  as  required 

7.3.4  Identify  potential  personnel  gaps  based  on 
detailed  system  designs 

7.3.5  Update  personnel  requirements 

7.3.6  Participate  in  detailed  system  design 
demonstrations 


Review  and  update  Training 
systems  plans 


7.4.1  Identify  potential  gaps  in  training  based  on 
detailed  system  designs 

7.4.2  Update  training  requirements 

7.4.3  Update  KSAs 

7.4.4  Update  media  analysis  of  training  tasks 

7.4.5  Review  selections  to  determine  if  still 
appropriate  based  on  current  system  design/ 

7.4.6  Participate  in  detailed  system  design 
demonstrations 

7.4.7  Design  training  solutions  (ILT  /  ICW / 
Embedded  training  (ACAT  1)/Performance  aids 

7.4.8  Develop  JQR/PQS  Sheet 

7.4.9  Develop  and  prototype  training  solutions 

7.4.10  Develop  an  assessment  plan  for  training 
solutions 


System  Hazard  Analysis  (SHA) 


7.5.1  Assess  detailed  system  design  to  ensure  they 
meet  SOH  requirements 

7.5.2  Provide  recommendations  for  redesign 
modification  and/or  develop  mitigation  strategies  for 
SOH  as  required 

7.5.3  Participate  in  detailed  system  design 
demonstrations 

7.5.4  Provide  recommendations  for 
redesign/modification  and/or  develop  mitigation 
strategies  for  Personnel  qualifications  as  required 

7.5.5  Refine/update  system  safety  analyses 

7.5.6  Finalize  hazard  analyses 
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Iterative  usability  testing 


7.6.1  Evaluations  of  Human  Perfomance  embedded 
in  prototype  system; 

7.6.2  Support  development  of  detailed  design 
specifications  for  HMI/workstations/workspaces/  HCI/ 
Facilities 

7.6.3  Iteratively  develop  and  review  prototypes, 
mockups,  screenshots,  drawings  and  simulations// 
Plan  and  conduct  cognitive  walkthroughs/  usability 
testing/  heuristic  evaluations/  workspace  evaluations 

7.6.4  Evaluate  initial  system  designs  and 
specifications  having  an  impact  on  human 
performance  and  safety 

7.6.5  Verify  detailed  system  designs 

7.6.6  Address  HFE  requirements 

7.6.7  Incorporate  HFE  design  standards 

7.6.8  Support  concept/mission  goals/objectives/ 
satisfy  user  needs  and  goals 

7.6.9  Mitigate  known  human  performance  risks 

7.6.10  Provide  recommendations  to  resolve  detailed 
system  design  deficiencies  impacting  users  and 
system  performance 

7.6.11  Update  predictive  models  based  on  outcomes 


7.7.1  Special  analysis  to  update  thresholds, 
objectives,  and  evolving  criteria  for  DT&E  and  OT&E 

7.7.2  Develop  a  user-centered  test  plan  to  evaluate 
initial  detailed  system  designs 

7.7.2.1  Include  operational  use  cases  and 
operationally  relevant  tasks 

7.7.3  Finalize  HFE  requirements 

7.7.4  Determine  assessment  methods 
Apply  analysis  findings  to  TEMP  7.7.5  Identify  metrics 

7.7.6  Develop  a  user-centered  test  plan  to  evaluate 
the  system  during  component  and  integration  testing 

7.7.7  Identify  scope  of  test  demonstra-  tions  and 
develop  procedures,  materials,  and  instruments 

7.7.8  Identify  high  risk  areas  for  testing 
operationally  relevant  use  cases  and  tasks 

7.7.9  Operationally  define  human  performance 
metrics/  validation  approaches  for  HFE  requirements 


114 


7.8 

Syncronize  with  DoDAF 

7.8.1  Provide  updated  information  to  RM  for 
continued  expansion  of  SVs  1,  2,  3,  6,  7,  8,  9, 
and  CONOPS; 

7.8.2  Continue  specification  of  HV-F-2  &  3,  and 
HV-G. 

7.8.3  Continue  populating  all  other  HVs  as 
system  evolves 

7.9 

Error  and  fault  analysis 

7.9.1  Analysis  to  cover  human  error  perormance, 
equipment  operability,  safety  procedures  and  error 
recovery  mechanisms. 

7.9.2  Conduct  trials  to  support  V&V,  and  acceptance 
against  HSI  requirements,  including  interoperability 

7.10. 

Begin  formal  development  of 
human-centered  source 
selection  criteria  for  RFPs  at 

MS  C 

7.10  Begin  formal  development  of  human-centered 
source  selection  criteria  for  RFPs  at  MS  C 

7.11 

Review  and  update  HSI  issues 
log 

7.11.1  Review  and  update  HSI  issues  log 

HRL 

Activity 

Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL8=OT&E 

Human 

Performance 

Parameters 

HRL-8  activities 

should  occur 
roughly  at  MS  C,  in 
support  of  LRIP 
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8.1 

HSI  Assessment  -  HRL8  (HSIA- 
8) 

8.1.1  Assessment  based  on  best  available  Solution 
Description  across  all  HSI  domains  to  define  level  of 
risk  in  each.  Each  domain  dictates  critical  items  of 
consideration  bearing  on  system  design  depending 
on  the  (initially  likely,  and  later  definite)  HTPs  in  a 
system  as  it  evolves.  Risk,  for  purposes  of  HRLs,  is 
mitigated  by  Professional  analytical  and  managerial 
activities  that  link  critical  items  of  consideration  to 
domain-specific  design  decisions,  including  well- 
informed  trade-offs.  Emphasis  at  this  stage  should 
be  optimizing  transfer  of  HSI  issues  and 
consideration  information  to  the  sustainment 
community,  most  specifically  logistics  center 
managers. 

8.1.2  HSI  WG  review  HSI  issues  log  and  mitigaton 
progress 

8.1.3  HSI  WG  specifies  and  recommends  risk  matrix 
outcomes  and  HRL  assignment  to  PM 

8.2 

Update/revise  HSIP 

8.2.1  Using  worksheet  as  described  in  HRL-1  to 
update  sequence  of  activities  remaining 

8.3a 

Update/revise  HFE  for  Design 
and  LCMP 

8.3.1  Evaluate  system  components  having  an  impact 
on  human  performance  and  safety 

8. 3.1.1  Identify  human  performance  risks 

8.3.1. 2  Provide  HFE  mitigation  strategies 

8.3.2  Provide  inputs  to  developmental  test  plans  to 
ensure  they  include  appropriate  human 
performance  assessments  for  integrated  system 
testing  to  address: 

8. 3.2.1  HFE  requirements 

8.3.2.1.1  User  interaction  with  HW/SW  interfaces, 

8. 3.2.1. 2  COTS  integration,  and 

8. 3.2.1. 3  operational  workflows 

8.3.2.2  HFE  high  risk  areas 

8. 3.2.3  Multiple  user  roles 

8.3.3  Collect  human  performance  data  during 
system  test  events 

8. 3.3.1  HFE  Requirements  Compliance  Assessments 

8. 3.3.2  HFE  System  Performance  Assessments 

8. 3.3.3  Usability  Testing 
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Update/revise  HFE  for  Design 
and  LCMP 


8.3.4  Document  HFE-related  system  performance 
issues  (i.e.,  PTRs) 

8.3.5  Verify  system  compliance  with  HFE 
requirements 

8.3.6  Analyze  results  from  HFE  testing 

8. 3. 6.1  Report  significance  of  findings  based  on  user 
impact  and  system  performance  risk 

8. 3. 6. 2  Incorporate  impact  assessment/  risk  analysis 
design  recommendations/  Human  Performance 
Mitigation  Strategies 

8.3.7  Support  the  review  and  development  of  job 
aids 

8.3.8  Participate  in  ECP  development 

8. 3. 8.1  Generate  HFE  related  ECPs 

8. 3. 8. 2  Review  ECPs  for  human  performance 
impacts 

8.3.9  Participate  in  /system  test  events 


Update/revise 

Manpower/Personnel  inputs 
for  Design  and  LCMP 


8.4.1  Perform  analysis  and  reporting  on  Manpower 
related  test  events,  data  collected,  and  findings 

8.4.2  Revise  the  ME  to  reflect  system  design 

8.4.2. 1  Provide  recommendations  based  on  findings 
to  support  manning  as  needed 

8.4.3  Perform  analysis  and  reporting  on  Personnel 
related  test  events,  data  collected,  and  findings 


Review  and  update  Training 
systems  for  Design  and  LCMP 


8.5.1  Assess  training  solutions  during  initial  system 
component  demonstrations 

8.5.2  Refine  training  solutions  based  on  findings 
from  initial  system  component  demonstrations/ 
Trainer  installations/  Equipment,  facility,  and  TTE 
plan 

8.5.3  Modify  ILE 

8.5.4  Provide  inputs  to  developmental  test  plans  to 
ensure  they  include  appropriate  assessments  of 
training  solutions  for  integrated  system  testing 

8.5.5  Participate  in  system  test  events 

8.5.6  Arrange  for  pilot/cadre  course  to  train  users 

8. 5.6.1  Assess  results  from  training  pilot  course 

8. 5.6.2  Compare  to  learning  objectives 

8.5.7  Revise  training  solutions  based  on  findings 
from  integrated  system  demonstrations  and  pilot 


8.5.8  Deliver  final  training  solution  for  use  in 
production  and  deployment  phase 
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8.6 

System  Hazard  Analysis 
(SHA)//Operating  and  Support 
Hazard  Analysis  (O&SHA) 

8.6.1  Participate  in  system  test  events 

8.6.2  Perform  analyses  and  reporting  on  SOH 
related  test  events,  data  collected,  and  findings 

8.6.3  Revise  the  System  Safety  Analyses  and  Hazard 
analyses  to  reflect  demonstrated  system  design 

8.6.4  Provide  recommendations  based  on  findings 
to  support  system  design  changes 

8.7 

Iterative  usability  testing 

8.7.1  Evaluations  of  Human  Perfomancce 
embedded  in  integrated  system; 

8.7.2  Update  predictive  models  based  on  outcomes 

8.8 

Apply  analysis  findings  to  TEMP 

8.8.1  Special  analysis  to  update  thresholds, 
objectives,  and  evolving  criteria  for  OT&E 

8.9 

Complete  formal  development 
of  human-centered  source 
selection  criteria  for  RFPs  at 

MS  C 

8.9.1  Update  draft  HP-related  source  selection 
criteria,  primarily  from  integrated  prototype  DT&E 
and  advanced  usability  testing 

8.10. 

Syncronize  with  DoDAF 

8.10.1  Provide  updated  information  to  RM  for 
continued  expansion  of  SVs  1,  2,  3,  6,  7,  8,  9,  and 
CONOPS; 

8.10.2  Continue  specification  of  HV-F-2  &  3,  and  HV- 
G. 

8.10.3  Continue  populating  all  other  HVs  as  system 
evolves 

8.11. 


Review  and  update  HSI  issues  . 

8.11.1  Review  and  update  HSI  issues  log 


HRL 

Activity 

Sub-Activity  Detail 

Acquisition 
&Sustainment 
Toolkit  Linkages 

Integration 
Notes  and 

Comments 

Target 
Documents 
(e.g.  ICD  or 
RFP) 

Action  OPR 
(generally 
within  HSI 

WG) 

References 

Products  (of 
Activity) 

Rough  Cost 
Estimate 

HRL9= 

Sustainment: 
Initiation  of 
Capability  Gap 
Feedback  Cycle 

HRL  9  Hand-off  transition  from 
Acquisition  PM  to 
Logistics/sustainment  PM  as 
keeper  of  HSI 

Plan/documentation.  Shred 
copy  to  next  block  Acq  PM. 

Emphasis  at  this  stage 
should  be  optimizing 
transfer  of  HSI  issues 
and  consideration 
information  to  the 
sustainment 
community,  most 
specifically  logistics 
center  managers. 

9.1 

HSI  Assessment  -  HRL9  (HSIA- 

9) 

9.1.1  Assessment  based  on  best  available  Solution 
Description  across  all  HSI  domains  to  define  level  of 
risk  in  each.  Each  domain  dictates  critical  items  of 
consideration  bearing  on  system  design  depending 
on  the  (initially  likely,  and  later  definite)  HTPs  in  a 
system  as  it  evolves.  Risk,  for  purposes  of  HRLs,  is 
mitigated  by  Professional  analytical  and  managerial 
activities  that  link  critical  items  of  consideration  to 
domain-specific  design  decisions,  including  well- 
informed  trade-offs. 

9.1.2  HSI  WG  review  HSI  Issues  log  and  mitigaton 
progress 

9.1.3  HSI  WG  specifies  and  recommends  risk  matrix 
outcomes  and  HRL  assignment  to  PM 

9.1.4  Define  processand  responsibilities  for  drafting 
Program  Increment  HSI  Transition  Final  Report  for 
Logistics/  Sustainment  PM 

Transition  to 
Logistics/Sustainment 
Program  Management  is 
Emphasis  and  Keystone 
to  HSI  success 
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HFE  10  T&E  support 


9.2  Review  verify  and  develop  HFE  installation 
criteria/ 

9.2.1. 1  Provide  recommendations  for  improvement 

9.2.1. 2  Consider  human  performance,  workload, 
and  safety/  Identify  gaps  or  potential  installation 
problems 

9.2.1. 3  Perform  analysis  of  LRIP  deficiencies  to 
Identify  human  performance  risks 

9.2.1.4  Provide  HFE  mitigation  strategies 

9.2.2  Develop  a  user-centered  assessment  plan  to 
support  OT&E  considering  performance  quality/ 
compatibility/  interoperability/  system  capabilities/ 

9.2.3  Participate  in  IOT&E 

9.2.3.1  Validate  HFE-based  system  changes 

9.2.3.2  Collect  user  feedback  and  human 
performance  data 

9.2.3.3  Report  significance  of  findings  based  on  user 
impact  and  system  performance  risk 

9.2.4  Review  HSI  ISSUE  reports  and  discuss  HSI 
migations/tradeoffs 

9.2.5  Verify  and  validate  final  HFE-related 
production  items 


HFE  FO  T&E  support 


9.3.1  Collect  and  analyze  user  feedback  and  service 
use  data  from  the  post-deployed  system 

9.3.2  Analyze  known  system  performance 
issues/program  trouble  reports 

9.3.3  Identify  HFE  lessons  learned  and  life  cycle 
improvements 

9.3.4  Provide  ECP  recommendations 

9.3.5  Provide  mitigation  strategies  for  outstanding 
HFE  issues  and  risks 

9.3.6  Develop  a  user-centered  assessment  plan  to 
support  FOT&E  considering  performance/  quality/ 
compatibility/  interoperability/  system  capabilities 

9.3.7  Participate  in  FOT&E 

9.3.7.1  Analyze  results  from  FOT&E  to  assess  and 
redesign  aspects  of  the  system/  workstations/ 
workspaces/  allocation  of  functions/  procedures/  job 
aids /  Human-Machine  Interfaces/  Human-Computer 
Interfaces 

9.3.8  Develop  an  operations  and  support  plan  to 
monitor  and  assess  the  system  and  collect  feedback 
from  users 
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HFE  FOC  support  and 
Logistics/Sustainment 
Transition 


9.4.1  Identify/document  HFE  lessons  learned  and 
lifecycle  improvements  from  fielded  system  and 
feed  forward 

9.4.2  Collect  feedback  from  operational  users  across 
multiple  platforms  and  commands 

9.4.3  Perform  a  system  assessment  to  identify 
capability  gaps 

9.4.4  Document  system  areas  that  have  been 
significantly  impacted  by  HFE  including: 
performance  risk  mitigation  safety  &  health,  quality 
of  life,  affordability 

9.4.4.1  Analyze  and  model  post-product 
improvements  for  the  next  incremental  build  of 
Human-Machine  Interfaces,  facilities, 
workstations/workspaces/  Human-Computer 
Interfaces 


HFE  FOC  support  and 
Logistics/Sustainment 
Transition 


9.4.4.2  Provide  HFE  based  recommendations  for 
system  modernization  and  increment  upgrades 

9.4.4.3  Identify  and  assess  design/engineering 
tradeoffs  for  modernization 

9.4.4.3.1  Identify  advanced  technologies  that  will 
have  a  significant  positive  impact  on  human 
performance 

9.4.5  Review  /HSI  ISSUE  /reports  and  /discuss  HSI 
/tradeoffs  from  an  HFE  design  impacts  perspective 

9.4.6  Participate  to  include  HFE  summary  in 
Program  Increment  HSI  Transition  Final  Report  for 
Logistics/  Sustainment  PM 


9.5.1  Transition  Post-Fielding  Training  Evaluation 
Analysis  (PFTEA).  Validation  of  system  training  in 
operational  setting  and  formal  certification  for  FOC 

9.5.2  Conduct  a  training  effectiveness 
evaluation//  identify  training  deficiencies  and  root 


Training  FOC  support  and 

Logistics/Sustainment 

Transition 


9.5.3  Provide  recommendations  to  correct  training 
deficiencies  /  incorporate  representative  mission 
scenarios 

9.5.4  Assess  human  performance  and  effectiveness 
of  training  transfer 

9.9.5  Redesign  training  procedures  and  decision 
aids 

9.5.6  Identify/document  training  lessons  learned 
and  feed  forward 

9.5.7  Incorporate  findings  into  Program  Increment 
HSI  Transition  Final  Report  for  Logistics/Sustainment 
PM 


The  PFTEA  is  required 
to  validate  institutional 
training  to  ensure 
training  requirements 
are  met. 
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9.6 

Manpower/Personnel  FOC 
support  and 
Logistics/Sustainment 

Transition 

9.6.1  Verify  ME  survey 

9.6.2  Verify  system  installations  meet  Manpower 
requirements 

9.6.3  Assess  effects  of  equipment  installations  on 
workload 

9.6.4  Identify/document  manpower  lessons  learned 
and  feed  forward 

9.6.5  Identify  and  document  areas  where 

Manpower  has  had  a  significant  impact 

9.6.6  Provide  modernization  /recom-mendations 
that  would  optimize  manpower 

9.6.7  Collect  feedback  from  users  to 
identify/document  personnel  lessons  learned  and 
feed  forward 

9.6.8  Review  HSI  issue  reports  and  discuss  HSI 
tradeoffs 

9.6.9  Incorporate  findings  into  Program  Increment 
HSI  Transition  Final  Report  for  Logistics/Sustainment 
PM 

9.7 

SOH  FOC  support  and 
Logistics/Sustainment 
TransitionOperating  and 

Support  Hazard  Analysis 
(O&SHA) 

9.7.1  Verify  system  installations  meet  SOH 
requirements 

9.7.2  Finalize  the  SOH  footprint  and  attributes. 
Identify/document  safety  and  occupational  health 
lessons  learned  and  feed  forward 

9.7.3  Complete  a  System  (fielded)  Safety  Analysis; 
include  /sustaining  hazard  analysis  for  the  fielded 
system  and  Incorporate  findings  into  Program 
Increment  HSI  Transition  Final  Report  for 
Logistics/Sustainment  PM 

9.8 

Syncronize  with  DoDAF 

9.8.1  Provide  updated  information  to  RM  for 
continued  expansion  of  SVs  1,  2,  3,  6,  7,  8,  9,  and 
CONOPS; 

9.8.2  Continue  specification  of  HV-F-2  &  3,  and  HV- 
G. 

9.8.3  Continue  populating  all  other  HVs  as  system 
evolves. 

9.8.4  Archive  as  part  of  Program  Increment  HSI  Final 
Report  for  Logistics/Sustainment  PM 

SVs:  system  interface 
description, 

communication,  matrix, 
information  exchange 
matrix,  evolution 
description,  technology 
forcast,  and 
performance  parameter 
matrix.  Training 
requirements  (HV-F-2), 
training  resources  (HV-F 
3),  and  metrics  (HV-G). 
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APPENDIX  G.  HRL  SUBJECT  MATTER  EXPERT  EVALUATION 

QUESTIONNAIRE 


ABOUT  THIS  REVIEW 


You  are  invited  to  participate  in  a  Subject  Matter  Expert  (SME)  review  of  the  evolving 
concept  of  the  Human  Readiness  Level  (HRL).  As  this  is  the  first  iteration  of  the  HRL 
framework,  your  valuable  input  and  feedback  will  be  used  to  refine  and  ultimately 
advance  its  ongoing  definition  and  validation. 

Please  note  that  all  data  collected  in  this  review  will  be  kept  strictly  confidential.  Your 
participation  in  the  review  and  your  responses  to  the  review  items  will  not  be  disclosed. 
Additionally,  no  Personal  Identifying  Infonnation  (PII)  will  be  collected  in  this  review. 


HRL  BACKGROUND  &  FRAMEWORK  STRUCTURE 


In  order  to  effectively  translate  capability  needs  and  technology  opportunities  into  stable, 
affordable,  and  well-managed  acquisition  programs,  proper  measurement  and 
management  of  technology  maturity  must  occur.  The  Technology  Readiness  Level 
(TRL)  has  proven  to  be  the  tool  and  process  of  choice  for  assessing  the  maturity  of 
developing  technologies.  Yet,  due  mainly  to  its  intended  focus,  the  TRL  has  proven 
inappropriate  to  consistently  capturing  the  human-related  attributes  of  technology  and 
their  association  with  technology  readiness.  It  is  in  response  to  this  recognized  issue  that 
the  Human  Readiness  Level  (HRL)  framework  was  created. 

The  intended  purposes  of  HRLs  are  to: 

1.  directly  support  the  S&T  community  and  Resource  and  Program  Managers  by 
providing  a  risk  management  structure  and  process  that  is  similar  to  the  TRL  structure 
and  process,  but  that  explicitly  links  technology  development  to  the  effective  design  and 
specification  of  human-centered  systems  in  Defense  Acquisition; 

2.  directly  support  the  Requirements  and  Program  Managers  by  providing  a  potentially 
exhaustive  list  of  candidate  activities  to  be  tailored  for  each  program  to  populate  HSI 
Planning  with  data  collection  and  analytical  activities  needed  to  inform  requirements, 
acquisition,  and  sustainment  documents  of  record;  and, 

3.  support  the  policy-driven  mandate  to  implement  HSI  by  identifying  the  most  critical 
HSI-specific  data  collection/analysis  activities  as  factors  and  elements  in  cost  estimation 
during  the  Analysis  of  Alternatives  process;  this  will  “fund  the  requirement”  by  directly 
informing  PPBS  insertion  to  complement  traditional  Work  Breakdown  Structure-based 
program  factors  and  elements. 

To  accomplish  these  ends,  the  HRL  framework  must  provide  a  time/milestone-sensitive 
roadmap  of  activities  (explicitly  linked  to  existing  Acquisition  and  Sustainment  Toolkits, 
for  example,  and  related  guidance  and  standards)  that:  a)  detail  critical  organizational 
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milestones  that  ensure  functional  commitment  to  Human  Systems  Integration  (HSI);  and 
b)  define  HSI  domain-sensitive  data  collection  and  analysis  activities.  The  primary 
measure  of  HSI  risk  and  maturity  level  is  based  on  the  execution  of  those  milestone- 
sensitive  management  and  analytical  activities. 

At  the  core  of  the  HRL  concept  is  the  activities  list,  the  central  topic  for  this  review.  In  a 
later  stage  of  HRL  development,  an  explicit  process  will  be  defined  for  risk  management 
and  reporting.  The  process  will  define  how  to  assign  each  HRL  as  an  assessed  program 
achievement  value,  a  risk  management  metric.  The  metric  will  depend  upon  the 
determination  of  a  scale  value  from  1  (HSI  baselining  &  commitment)  to  9  ( capability 
gap  feedback  cycle).  A  program  determined  to  be  at  HRL-1  would  have  achieved  the 
lowest  level  of  HSI  readiness,  where  the  initial  activation  of  HSI  commitment  and 
processes  occurs.  By  the  time  the  program  has  reached  HRL-9,  it  will  have  progressed 
through  significant  HSI  analyses,  requirement  threshold  refinements,  field  validations  of 
human  perfonnance  prototypes,  and  extensive  developmental  and  operational  Test  and 
Evaluation  (T&E)  of  human  performance  parameters.  As  with  TRLs,  what  constitutes  an 
acceptable  and  appropriate  Human  Readiness  Level  (and  so,  level  of  programmatic  risk) 
will  depend  upon  a  large  number  of  program-specific  details,  most  critically  linked  to 
where  a  program  of  record  is  in  its  life  cycle. 

By  structuring  appropriate  sequencing  of  HSI  activities  and  tracking  the  progress  of  HSI 
planning  and  execution,  the  HRL  distinguishes  itself  as  a  significant  risk  management 
tool  for  process  owners  and  decision  makers  in  Defense  Acquisition.  The  following  is 
the  first  evaluation  of  the  HRL  framework’s  worth  and  usability.  Based  on  your 
feedback,  our  plan  is  to  develop  an  updated  draft  of  the  candidate  list  and  framework  to 
form  the  basis  for  a  series  of  developmental  workshops  to  “fill  in  the  matrix”  and  develop 
this  into  as  practical  a  tool  as  is  possible.  Your  inputs  will  be  critical  to  this  effort. 


INSTRUCTIONS 


The  first  worksheet  (labeled:  HRL  Framework  Draft!)  in  the  attached  Excel  Spreadsheet 
contains  the  content  for  your  review.  Columns  A,  B  &  C  contain  HSI  candidate 
activities,  while  columns  D  through  J  contain  proposed  column  headings  for  future 
HRL  framework  efforts.  Only  for  the  first  three  columns  in  the  intended  framework 
have  the  cells  been  filled  in  completely. 

The  following  items  are  designed  to  gain  feedback  as  to  the  accuracy,  ease  of  use,  and 
completeness  of  the  HRL  framework  activities  lists.  The  items  are  organized  into  five 
separate  groups.  The  first  four  grouping  of  statements  reflect  the  HRL’s  recommended 
milestone  progression  in  the  Defense  Acquisition  Life  Cycle.  For  instance,  HRL-2  is  the 
criterion  threshold  to  be  met  at  Milestone  A,  therefore  the  first  set  of  items  pertains  to 
HRLs  1  and  2.  The  last  set  of  items  found  in  the  fifth  group  involves  the  proposed 
categories  for  future  HRL  framework  efforts. 
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Using  the  attached  draft  of  the  HRL  Framework,  for  each  item  below,  circle  or  underline 
the  response  that  best  describes  your  feelings.  Please  add  comments  as  needed. 


URLs  1/2 

*HRL-2  should  be  achieved  prior  to  Milestone  A  approval. 

HRL  Descriptions  -  Column  A _ 

1 .  The  HRL  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

2.  The  HRL  descriptions  reflect  appropriate  timing  within  the  acquisition  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

3.  No  critical  information  is  missing  from  the  HRL  descriptions. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

Activity  -  Column  B _ 

4.  The  activities  and  their  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

5.  The  activities  are  appropriately  timed  within  the  acquisition  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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6.  No  critical  information  is  missing  from  the  activities  and  their  descriptions. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

Sub-Activity  Details  -  Column  C _ 

7.  The  HRL  sub-activity  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

8.  The  HRL  sub-activity  details  reflect  appropriate  timing  within  the  acquisition  life 
cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

9.  No  critical  items  of  activity  are  missing  from  the  sub-activities  list. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

HRLs  3/475/6 

*HRL  -  6  should  be  achieved  prior  to  Milestone  B  approval. 

HRL  Descriptions  -  Column  A _ 

10.  The  HRL  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

1 1 .  The  HRL  descriptions  reflect  appropriate  timing  within  the  acquisition  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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12.  No  critical  information  is  missing  from  the  HRL  descriptions. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 
Comments 


Activity:  Column  B _ 

13.  The  activities  and  their  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 
Comments 

14.  The  activities  are  appropriately  timed  within  the  acquisition  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

15.  No  critical  infonnation  is  missing  from  the  activities  and  their  descriptions. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

Sub-Activity  Details:  Column  C _ 

16.  The  HRL  sub-activity  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

17.  The  HRL  sub-activity  details  reflect  appropriate  timing  within  the  acquisition  life 
cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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18.  No  critical  items  of  activity  are  missing  from  the  sub-activities  list. 
Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 
Comments 


URLs  7/8 

*HRL  -  8  should  be  achieved  prior  to  Milestone  C  approval. 

HRL  Descriptions:  Column  A _ 

19.  The  HRL  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

20.  The  HRL  descriptions  reflect  appropriate  timing  within  the  acquisition  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

21.  No  critical  information  is  missing  from  the  HRL  descriptions. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

Activity;  Column  B _ 

22.  The  activities  and  their  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

23.  The  activities  are  appropriately  timed  within  the  acquisition  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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24.  No  critical  information  is  missing  from  the  activities  and  their  descriptions. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

Sub-Activity  Details:  Column  C _ 

25.  The  HRL  sub-activity  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

26.  The  HRL  sub-activity  details  reflect  appropriate  timing  within  the  acquisition  life 
cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

27.  No  critical  items  of  activity  are  missing  from  the  sub-activities  list. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

HRL  9 

*HRL  -  9  represents  the  highest  HRL  to  be  achieved  and  signifies  the  initiation  of  the 
capability  gap  feedback  cycle. 

HRL  Descriptions:  Column  A _ 

28.  The  HRL  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

29.  The  HRL  descriptions  reflect  appropriate  timing  within  the  acquisition  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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30.  No  critical  information  is  missing  from  the  HRL  descriptions. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

Activity;  Column  B _ 

3 1 .  The  activities  and  their  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

32.  The  activities  are  appropriately  timed  within  the  acquisition  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

33.  No  critical  infonnation  is  missing  from  the  activities  and  their  descriptions. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

Sub-Activity  Details:  Column  C _ 

34.  The  HRL  sub-activity  descriptions  are  clearly  understood. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

35.  The  HRL  sub-activity  details  reflect  appropriate  timing  within  the  acquisition  life 
cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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36.  No  critical  items  of  activity  are  missing  from  the  sub-activities  list. 
Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 
Comments 


Proposed  Categories  (Columns)  for  Future  HRL  Framework  Efforts 

A  &  S  Toolkit:  Column  D _ 

37.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

38.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 


Integration  Notes  and  Comments:  Column  E _ 

39.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

40.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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Target  Documents:  Column  F _ 

41.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 


42.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 


Action  OPR:  Column  G _ 

43.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 


44.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 


References:  Column  H _ 

45.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 


132 


46.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 


Products  of  Activity;  Column  I _ 

47.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 


48.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

Rough  Cost  Estimate  (person  days  and  $):  Column  J _ 

49.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

50.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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51.  You  would  recommend  no  additional  column  topics  to  populate  the  matrix. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments;  please  list  any  additions  you  would  recommend  and  your  rationale,  if 
appropriate 

#5 1  is  the  last  rating  item  summarizing  your  review.  Please  add  any  comments,  issues,  or 
concerns  below.  Thank  you  for  your  time  and  effort  in  support  of  this  work  in  progress. 
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APPENDIX  H.  HRL  SUBJECT  MATTER  EXPERT  CASE  STUDY 

QUESTIONNAIRE 


ABOUT  THIS  REVIEW 


You  are  invited  as  key  HSI  practitioners  in  a  program  office  to  participate  in  a  review  of 
activity  lists  and  matrix  headings  contained  in  the  first  installment  of  the  evolving 
concept  of  the  Human  Readiness  Level  (HRL).  As  this  is  the  first  iteration  of  the  HRL 
framework,  your  valuable  input  and  feedback  will  be  used  to  refine  and  ultimately 
advance  its  ongoing  definition  and  validation. 

Please  note  that  all  data  collected  in  this  review  will  be  kept  strictly  confidential.  Your 
participation  in  the  review  and  your  responses  to  the  review  items  will  not  be  disclosed. 
Additionally,  no  Personal  Identifying  fnfonnation  (PH)  will  be  collected  in  this  review. 


HRL  BACKGROUND  &  FRAMEWORK  STRUCTURE 


In  order  to  effectively  translate  capability  needs  and  technology  opportunities  into  stable, 
affordable,  and  well-managed  acquisition  programs,  proper  measurement  and 
management  of  technology  maturity  must  occur.  The  Technology  Readiness  Level 
(TRL)  has  proven  to  be  the  tool  and  process  of  choice  for  assessing  the  maturity  of 
developing  technologies.  Yet,  due  mainly  to  its  intended  focus,  the  TRL  has  proven 
inappropriate  to  consistently  capturing  the  human-related  attributes  of  technology  and 
their  association  with  technology  readiness.  It  is  in  response  to  this  recognized  issue  that 
the  Human  Readiness  Level  (HRL)  framework  was  created. 

The  intended  purposes  of  HRLs  are  to: 

1.  directly  support  the  S&T  community  and  Resource  and  Program  Managers  by 
providing  a  risk  management  structure  and  process  that  is  similar  to  the  TRL  structure 
and  process,  but  that  explicitly  links  technology  development  to  the  effective  design  and 
specification  of  human-centered  systems  in  Defense  Acquisition; 

2.  directly  support  Requirements  and  Program  Managers  by  providing  a  potentially 
exhaustive  list  of  candidate  HSI-specific  data  collection  and  analytical  activities  to  be 
tailored  for  each  program  to  populate  HSI  Planning  to  inform  requirements,  acquisition, 
and  sustainment  documents  of  record;  and, 

3.  support  the  policy-driven  mandate  to  implement  HSI  by  identifying  the  most  critical 
HSI-specific  data  collection/analysis  activities  as  factors  and  elements  in  cost  estimation 
during  the  Analysis  of  Alternatives  process;  this  will  “fund  the  requirement”  by  directly 
informing  Planning,  Programming,  and  Budgeting  System  (PPBS)  insertion  to 
complement  traditional  Work  Breakdown  Structure-based  program  factors  and  elements. 
To  accomplish  these  ends,  the  HRL  framework  must  provide  a  time/milestone-sensitive 
roadmap  of  activities  (explicitly  linked  to  existing  System  Engineering  processes, 
Acquisition  and  Sustainment  Toolkits,  for  example,  and  related  guidance  and  standards) 
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that:  a)  detail  critical  organizational  milestones  that  ensure  functional  commitment  to 
Human  Systems  Integration  (HSI);  and  b)  define  HSI  domain-sensitive  data  collection 
and  analysis  activities  and  a  clear  process  for  their  management.  The  primary  measure  of 
HSI  risk  and  maturity  level  will  be  based  on  the  execution  of  those  milestone-sensitive 
management  and  analytical  activities. 

At  the  core  of  the  HRL  concept  is  the  activities  list,  the  central  topic  for  this  review.  In  a 
later  stage  of  HRL  development,  an  explicit  process  will  be  defined  for  risk  management 
and  reporting.  The  process  will  define  how  to  assign  each  HRL  as  an  assessed  program 
achievement  value,  a  risk  management  metric.  The  metric  will  depend  upon  the 
determination  of  a  scale  value  from  1  (HSI  baselining  &  commitment)  to  9  ( capability 
gap  feedback  cycle).  A  program  determined  to  be  at  HRL-1  would  have  achieved  the 
lowest  level  of  HSI  readiness,  where  the  initial  activation  of  HSI  commitment  and 
processes  occurs.  By  the  time  the  program  has  reached  HRL-9,  it  will  have  progressed 
through  significant  HSI  analyses,  requirement  threshold  refinements,  field  validations  of 
human  perfonnance  prototypes,  and  extensive  developmental  and  operational  Test  and 
Evaluation  (T&E)  of  human  performance  parameters.  As  with  TRLs,  what  constitutes  an 
acceptable  and  appropriate  Human  Readiness  Level  (and  so,  level  of  programmatic  risk) 
will  depend  upon  a  large  number  of  program-specific  details,  most  critically  linked  to 
where  a  program  of  record  is  in  its  life  cycle. 

The  key  to  a  successful  HSI  acquisition  strategy  is  having  an  accurate  HSI  Plan 
(HSIP)  in  place.  The  following  is  the  first  evaluation  of  the  HRL  framework’s  ability 
to  directly  support  the  creation  and  execution  of  an  HSIP.  Based  on  your  feedback, 
our  plan  is  to  develop  an  updated  draft  of  the  activity  candidate  list  and  framework  to 
form  the  basis  for  a  series  of  developmental  workshops  to  “fill  in  the  matrix”  and  develop 
this  into  as  practical  a  tool  as  is  possible.  Your  inputs  will  be  critical  to  this  effort. 


INSTRUCTIONS 


The  first  worksheet  (labeled:  HRL  Framework  Draft!)  in  the  attached  Excel  Spreadsheet 
contains  the  content  for  your  review.  Columns  A,  B  &  C  contain  HSI  candidate 
activities,  while  columns  D  through  J  contain  proposed  column  headings  for  future 
HRL  framework  efforts.  Only  for  the  first  three  columns  in  the  intended  framework 
have  the  cells  been  filled  in  completely. 

The  following  items  are  designed  to  gain  feedback  as  to  the  ease  of  use,  and  completeness 
of  the  HRL  framework  when  used  to  support  the  creation  and  execution  of  an  HSIP.  For 
each  item  below,  circle  or  underline  the  response  that  best  describes  your  feelings.  Please 
add  comments  as  needed. 
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1.  The  milestone-sensitive  HSI  activities  listed  and  the  HRL  matrix  (once  completed) 
will  provide  an  effective  framework  for  an  HSI  working  group  to  tailor  for 
comprehensive  HSI  planning. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

2.  The  HRL  framework  contains  all  of  the  appropriate  HSI  activities  that  would  be 
necessary  to  manage,  create,  and  sustain  an  HSIP  throughout  a  program’s  life  cycle. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

3.  Using  the  HRL  framework  as  an  HSI  maturity  metric  can  benefit  HSI  risk 
identification  and  management  within  the  HSIP. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 

4.  The  HRL  framework  facilitates  easier  HSI  issue-tracking  and  HSI  execution 
accountability. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments 
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5.  The  column  headings  in  the  spreadsheet  (after  columns  A,  B  &  C  being  reviewed 
above)  represent  a  complete  list  of  the  key  areas  of  information  needed  to  perform 
effective  HSI  Planning. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 
Comments  (please  recommend  any  additional  headings  you  think  are  needed) 


6.  The  column  headings  in  the  spreadsheet  (after  columns  A,  B  &  C  being  reviewed 
above)  represent  the  most  relevant  list  of  the  key  areas  of  information  needed  to  perform 
effective  HSI  Planning. 

Strongly  Agree  Agree  Neutral  Disagree  Strongly  Disagree 

Comments  (please  identify  headings  you  think  might  be  eliminated  as  redundant  or 
irrelevant) 


7.  The  following  section  is  designed  to  gain  feedback  as  to  the  relative  value/importance 
of  the  current  (columns  A,  B  &  C)  and  proposed  categories  (columns  D  -  J)  contained  in 
the  HRL  framework  when  being  used  to  support  an  HSIP.  Please  assign  a  rank  to  each  of 
the  ten  categories  listed  below  in  order  of  value/importance  to  HSI  Planning.  (“1”  is  most 
valuable/important;  tied  values  in  the  ranking  are  acceptable;  please  comment  where 
appropriate;  and,  especially,  specify  any  additional  columns  you  believe  need  to  be  added 
or  any  current  heading(s)  that  you  feel  should  be  eliminated) _ 


a.  HRL  - 


b.  Activity  - 


c.  Sub  Activity  Detail  - 


d.  Acquisition  &  Sustainment  (A  &  S)  Toolkit  Linkages  - 


e.  Integration  Notes  &  Comments  - 
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f.  Target  Documents  - 


g.  Action  OPR  - 


h.  References  - 


i.  Products  (of  Activity)  - 


j.  Rough  Cost  Estimate  - 


7.  j.  is  the  last  rating  item  summarizing  your  review.  Please  add  any  additional 
comments,  issues,  or  concerns  below.  Thank  you  for  your  time  and  effort  in  support  of 
this  work  in  progress. 
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APPENDIX  I.  HRL  SUBJECT  MATTER  EXPERT  EVALUATION 

COMMENTS 


URLs  1/2 

*HRL-2  should  be  achieved  prior  to  Milestone  A  approval. 


HRL  Descriptions  -  Column  A _ 

1 .  The  HRL  descriptions  are  clearly  understood. 

“Could  be  better,  but  not  bad ” 

“The  description  for  HRL  1  is  rather  clear  and  straight  forward.  However  for  HRL  2, 
reference  to  “component  technology  development’’  is  not  real  clear.  Since  HRL  2  is  in 
support  of  MS  A  and  subsequent  Technology  Development  Phase,  suggest  something  like 
“HSI  analysis  suite  in  support  of  AoA  ’’  or  “HSI  analysis  suite  in  support  of  MS  A  ”  or 
“HSI  analysis  suite  in  support  of  proceeding  to  Technology  Development  Phase.  ” 

“1  is  relatively  clear,  but  the  only  issue  is  this  can  be  done  at  any  phase  based  on  when 
HSI  involvement  is  initiated.  I  understand  that  the  intent  is  to  begin  involvement  pre  MS 
A,  but  even  if  you  don  7  this  activity  can  be  accomplished  at  any  point  HSI  gets  involved. 

2  is  somewhat  less  clear  to  me,  I’m  not  sure  what  the  suite  is  referring  to,  is  it  a  toolkit  or 
checklist?  It  may  become  more  clear  as  I  read  through  the  information.  ’’ 


2.  The  HRL  descriptions  reflect  appropriate  timing  within  the  acquisition  life  cycle. 

“ Getting  a  commitmen  t  and  actually  getting  engaged  will  be  very  hard  prior  to  having  an 
established  program.  Granted  -  ideally  this  should  be  engaged  with  DP  and  CCTD,  etc. 
How  much  you  can  actually  accomplish  before  MS  B  will  be  challenging.  Being 
informed  by  CBA  and  CBP  and  other  analyses  may  provide  help.  ” 

“Timing  within  acquisition  cycle  cannot  be  ascertained  from  Column  A  descriptions.  ” 

“Strongly  agreed,  it  is  very  important  to  begin  as  early  as  possible  in  the  process.  Only 
question  is  what  happens  when  you  ’re  starting  after  MSA?  Do  you  still  go  through  these 
or  should  they  have  already  been  done?  It  may  become  OBE  in  the  future,  but  I’m 
thinking  about  programs  that  are  currently  in  process.  ” 
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3.  No  critical  information  is  missing  from  the  HRL  descriptions. 

“Appears  to  be  comprehensive.  Need  more  time  to  review.  ” 

“The  description  for  HRL  2  does  not  relay  ultimate  performance  objective.  As  read,  the 
objective  for  HRL  2  is  to  keep  HSI  practitioners  employed.  Besides  shortfalls  mentioned 
above,  descriptors  should  describe  the  ultimate  outcome  or  goal  for  the  HRL.  ” 

“I  would  like  to  understand  what  the  state  is  in  HRL  2.  I  know  you  ’re  planning  on  doing 
this  in  the  future,  but  identify  where  each  HRL  is  tied  to  TRLs  and  MRLs  to  better  support 
the  technology  being  developed,  since  HRLs  are  more  process  focused  and  TRLs/MRLs 
are  product  focused.  ” 


Activity  -  Column  B _ 

4.  The  activities  and  their  descriptions  are  clearly  understood. 

“References  to  “ design ”  and  “system”  are  out  of  place  this  early  in  the  acquisition 
process  where  you  ’re  considering  concept  alternatives  (materiel  and  non-materiel) . 
Some  of  the  activities  have  same  title  (ex  11.  a  and  11.  b).  Recommend  activity  state  what 
it  is  that  is  to  be  performed.  Begin  each  activity  with  “Perform”,  “Conduct”, 
“Generate”,  “List”,  etc.” 

“1  -  Most  of  them  are  easily  understood  but  I’m  not  sure  how  well  they  will  be 
understood  by  non-HSI  individuals.  1.2  and  think  you  may  want  to  differentiate  1.11a  and 
b  and  1.12a  and  b  more  the  detail  shouldn’t  be  different  if  it’s  the  same  activity  at  the 
same  stage.  ” 

“Activity  listing  format  will  make  it  hard  for  future  development  of  cost  elements.  Can 
have  a  descriptor  column  for  detail.  ” 


5.  The  activities  are  appropriately  timed  within  the  acquisition  life  cycle. 

“Timing  for  phase  adequate.  ” 

“References  to  “ design ”  and  “system”  are  out  of  place  this  early  in  the  acquisition 
process  where  you  ’re  considering  concept  alternatives  (materiel  and  non-materiel).  ” 

“Some  of  the  documents  are  not  required  at  MS  A.  But  I  agree  that  the  other  efforts 
should  begin  at  this  stage  to  ensure  effective  HSI.  ” 

“These  timings  are  probably  program-dependent,  but  they  seem  reasonably  aligned  with 
the  DODI  5000.02  timeline,  but  I  am  unclear  as  to  what  processes  are  supporting  CJCSI 
3170  activities  that  proceed  the  acquisition  program.  Current  information  indicates  that 
HSI  SMEs  need  to  be  operating  in  support  of  potential  programs  to  bring  forward 
lessons  learned  and  early  risk  mitigation  issues  well  before  Milestone  A.  ” _ 


142 


6.  No  critical  information  is  missing  from  the  activities  and  their  descriptions. 

“Needs  further  scrutiny.  ” 

“We  don  7  know  what  we  don  7  know.  We  may  find  that  there  are  critical  drivers  that  are 
not  under  our  purview  or  within  our  awareness  on  any  given  program,  or  FoS/SoS.  This 
framework  will  probably  best  be  constructed  as  an  iterative  model  that  accommodates 
periodic  modifications  (additions  /  deletions).  ’’ 


Sub-Activity  Details  -  Column  C _ 

7.  The  HRL  sub-activity  descriptions  are  clearly  understood. 

“Under  2.4.5  -  don’t  understand  inclusion  of  reference  to  2.4.1  training  situation 
analysis  (TSA).  Either  numbering  is  wrong  or  information  has  been  inserted  in  the 
wrong  place.  ” 

“I  like  how  they’re  broken  into  steps.  Makes  them  easier  to  follow  and  accomplish  each 
activity.  ’’ 


8.  The  HRL  sub-activity  details  reflect  appropriate  timing  within  the  acquisition  life 
cycle. 

“As  mentioned  above  with  the  Activity  detail.  Some  of  the  documents/efforts  may  not  be 
in  the  appropriate  place  but  I  think  it  will  take  trying  it  out  on  a  program  to  really  see 
what  can  be  accomplished  pre  MS  A.  At  this  level  I  can  7  think  of  specifics  besides  some 
of  the  documents,  but  there  may  be  some  modeling  that  can  7  be  done  as  fully  as 
anticipated  per  the  descriptions.  ” 

“Probably  realistically  a  little  early  on  many  of  them.  May  start  before  A,  but  unlikely  to 
have  the  necessary  funding,  fidelity  and  support  necessary  to  take  all  the  tasks  to 
completion  or  any  real  depth.  ’’ 

“1.6.4  Difficult  to  estimate  cost  until  manpower  is  more  thoroughly  explored  as  in  1.8.6. 
What  is  the  difference  between  1.8.4  and  1.11.2?  ’’ 

“I  think  as  we  examine  the  data  that  will  populate  columns  d,  f  &  g  timing  will  become 
clearer.  There  are  activities  that  may  be  premature  other  than  a  outline  for  future  phases 
and  other  activities  that  must  be  done  very  early  in  the  JC1DS  process  or  completed  as 
continuous  underlying  HSI  S&T  activities  that  are  pulled  off  the  shelf  into  the  JCIDS 
process  in  order  to  effectively  impact  (timely  input,  already  accepted  HSI  requirements)  a 
specific  material  solution.  The  cycle  time  is  too  short  to  undertake  some  HSI 
analyses/activities  and  drive  requirements  &  functional  allocation  decisions.  ” 


143 


9.  No  critical  items  of  activity  are  missing  from  the  sub-activities  list. 

“Needs  further  scrutiny  —  appears  to  be  comprehensive.  ” 

“Unable  to  determine.  The  answer  will  become  more  apparent  over  time  in  application  of 
the  HRL  process.  ” 


URLs  3/475/6 

*HRL  -  6  should  be  achieved  prior  to  Milestone  B  approval. 


HRL  Descriptions  -  Column  A _ 

10.  The  HRL  descriptions  are  clearly  understood. 

“Reference  to  “ component  ”  in  HRLs  3&4  can  mislead  that  system  level  was  bypassed  or 
circumvented.  Not  clear  what  is  “touch  point  resolution  ”  in  HRLs  3  &  4.  ” 

“I’m  not  sure  that  the  prototypes  are  going  to  be  completely  operational  by  MS  B.  I 
know  the  intent  is  to  follow  along  with  the  TRLs  but  HRL  6  with  the  current  definition 
may  be  somewhat  optimistic.  ” 


1 1 .  The  HRL  descriptions  reflect  appropriate  timing  within  the  acquisition  life  cycle. 
“Timing  within  acquisition  cycle  cannot  be  ascertained  from  Column  A  descriptions.  ” 


12.  No  critical  information  is  missing  from  the  HRL  descriptions. 

“Needs  more  scrutiny.  ” 

“Descriptors  shoidd  better  distinguish  maturity  levels,  particularly  HRLs  3  &  4." 

“This  should  be  discussed  in  more  detail  at  the  workshop.  But  for  now  I  think  top  level  is 
well  defined.  ” 


Activity;  Column  B _ 

13.  The  activities  and  their  descriptions  are  clearly  understood. 

“Could provide  some  more  explanation  /  description.  ” 

“It  is  difficult  to  assess  sufficiency  of  activity  descriptions  when  it  is  unclear  when  the 
milestone  points  are  unclear.  ” 


144 


14.  The  activities  are  appropriately  timed  within  the  acquisition  life  cycle. 

“  ...don ’t  assume  program  start  at  MS  A  —  normally  not  formalized  till  much  later.  ” 

“Stating  HRLs  3-5  all  should  occur  “as  soon  as  possible  after  MS  A  ”  does  not  help 
discriminate  between  the  three.  Timing  for  activity  completion  should  be  more  defined. 
Recommend  tailoring  HRL-4  activities/attainment  by  SRR/RFP  completion.  Recommend 
tailoring  HRL-5  activities/attainment  by  source  selection/SFR  completion.  Recommend 
tailoring  HRL-6  activities/attainment  by  PDR  completion  (before  or  after  MS  B).  ” 


15.  No  critical  infonnation  is  missing  from  the  activities  and  their  descriptions. 
“Needs  more  scrutiny.  ” 


Sub-Activity  Details:  Column  C _ 

16.  The  HRL  sub-activity  descriptions  are  clearly  understood. 

“HTPs  not  well  understood  /  articulated  throughout.  ” 

“ Further  detail  needs  to  be  provided  for  some  domains.  I.E.  Safety  /  Occupational 
Health  Activities.  Yes  they  looked  at  but  actual  activity  is  not  the  domain.  ” 


17.  The  HRL  sub-activity  details  reflect  appropriate  timing  within  the  acquisition  life 
cycle. 


18.  No  critical  items  of  activity  are  missing  from  the  sub-activities  list. 
“Add  3. 6. 1.3  Inform  cost  estimate  from  manpower  estimate.  ’’ 


HRLs  7/8 

*HRL  -  8  should  be  achieved  prior  to  Milestone  C  approval. 


HRL  Descriptions;  Column  A _ 

19.  The  HRL  descriptions  are  clearly  understood. 


20.  The  HRL  descriptions  reflect  appropriate  timing  within  the  acquisition  life  cycle. 


21.  No  critical  information  is  missing  from  the  HRL  descriptions. 
“Need  to  incorporate  HSI  trade-offs  and  integration  of  the  domains.  ” 


145 


Activity;  Column  B _ 

22.  The  activities  and  their  descriptions  are  clearly  understood. 


23.  The  activities  are  appropriately  timed  within  the  acquisition  life  cycle. 

“The  timing  for  HRL-7  activities  seems  appropriate.  Difficult  to  ascertain  timing  for 
“update/revise  HFE  for  design  and  LCMP  ”  references.  Reference  to  source  selection  by 
TRIP  is  out  of  place.  Generally,  PM  may  exercise  option  on  the  existing  contract  for 
contractor  to  produce  LRIP  units.  ” 


24.  No  critical  infonnation  is  missing  from  the  activities  and  their  descriptions. 

“The  analysis  by  CDR for  HRL-7  needs  to  be  pretty  comprehensive  as  the  design  is  pretty 
much  locked  in  after  this  point  and  changes  become  difficult  to  influence/implement. 
Appears  activities  for  HRL-8  should  be  better  tailored  toward  what  resulted  from  DT&E, 
and  how  will  deficiencies  be  addressed  prior  to  OT&E.  ” 

“You  may  want  to  add  additional  information  about  DT&E  and  OT&E  participation  and 
the  DRs  associated  with  these  tests.  Also,  include  inputs/final  analysis  and  trade-offs 
important  to  design  reviews.  Some  of  these  may  be  based  on  financial  implications  but 
that  needs  to  be  identified.  Final  opportunity  to  influence  design.  ” 


Sub-Activity  Details:  Column  C _ 

25.  The  HRL  sub-activity  descriptions  are  clearly  understood. 

“Generally  understand  descriptions,  but  some  should  be  improved.  Activities  related  to 
“participate  ’’  are  rather  unclear.  ” 


26.  The  HRL  sub-activity  details  reflect  appropriate  timing  within  the  acquisition  life 
cycle. 

“The  timing  for  activities  generally  seem  appropriate.  References  to  providing  input  to 
development  test  plans  (8.3.2)  and  source  selection  criteria  (8.9.1)  are  too  late  in 
process.  ” 


27.  No  critical  items  of  activity  are  missing  from  the  sub-activities  list. 

“Need  to  find  a  way  to  take  findings  from  DT&E  and  inform  /  support  OT&E  test 
readiness  evaluation  /  report.  Almost  all  HRL  testing  can  be  accomplished  unobtrusively 
in  DT&E  and  in  enough  detail  to  know  what  to  expect  in  OT&E.  HRL  determination  in 
OT&E  will  have  to  be  a  natural  outcome  from  the  actual  tests  -  it  must  be  unobtrusive. 
Also  may  want  to  get  involved  in  OAs  and  EOAs  to  refine  HRL  6  info  in  preparation  for 
final  DT&E  parameters.  ” 
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URL  9 

*HRL  -  9  represents  the  highest  HRL  to  be  achieved  and  signifies  the  initiation  of  the 
capability  gap  feedback  cycle. 


HRL  Descriptions:  Column  A _ 

28.  The  HRL  descriptions  are  clearly  understood. 

"Recommend  this  HRL/ description  be  aligned  with  Post  Implementation  Review.  ” 


29.  The  HRL  descriptions  reflect  appropriate  timing  within  the  acquisition  life  cycle. 


30.  No  critical  information  is  missing  from  the  HRL  descriptions. 


Activity;  Column  B _ 

3 1 .  The  activities  and  their  descriptions  are  clearly  understood. 


32.  The  activities  are  appropriately  timed  within  the  acquisition  life  cycle. 


33.  No  critical  infonnation  is  missing  from  the  activities  and  their  descriptions. 


Sub-Activity  Details:  Column  C _ 

34.  The  HRL  sub-activity  descriptions  are  clearly  understood. 


35.  The  HRL  sub-activity  details  reflect  appropriate  timing  within  the  acquisition  life 
cycle. 


36.  No  critical  items  of  activity  are  missing  from  the  sub-activities  list. 

"It  is  important  to  include  feedback  throughout  the  entire  acq  process,  not  just  during 
HRL  9.” 
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Proposed  Categories  (Columns)  for  Future  HRL  Framework  Efforts 


Acquisition  &  Sustainment  Toolkit  Linkages:  Column  D _ 

37.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

“It  will  likely  be  critical  for  more  than  HRL  9  only.  Since  Logistics  considerations 
should,  like  manpower  and  training  issues,  be  dealt  with  early,  it  is  not  unlikely  that 
much  lower  HRL  levels  could  be  influenced  by  the  ASTK.  ” 


38.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 


Integration  Notes  and  Comments:  Column  E _ 

39.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

“Agreed,  additional  details  will  be  especially  useful  for  individuals  tasked  with  HSI  that 
are  not  very  familiar  with  it  or  with  the  acquisition  process.  ” 


40.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 


Target  Documents:  Column  F _ 

41.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

“Agreed,  individuals  can  go  look  at  those  documents  for  additional  detail  on  program.  ” 

“Recommend  considering  list  of  target  documents  for  NEXT  phase  here  as  well  to 
highlight  upcoming  tasks  for  planning  purposes.  ” 


42.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 
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Action  OPR:  Column  G _ 

43.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

" There  should  be  a  program  office  OPR,  a  Sponsor /MAJCOM  OPR,  and  an  HSI  OPR.  " 


44.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

“ Especially  when  individuals  move  on,  it  is  nice  to  have  a  POC  to  go  back  to  or  identify 
that  the  work  has  been  in  process.  ” 


References:  Column  H _ 

45.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 


46.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

“Will  provide  information  to  individuals  performing  the  tasks  on  exactly  what  they  need 
to  do  and  where  to  go  to  find  information.  ” 

“Consider  hyper  linking  or  a  separate  reference  database/ dictionary.  ” 


Products  of  Activity:  Column  I _ 

47.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

“Strongly  agree...  for  building  off  of  effort  in  next  HRL  or  activity  ” 


48.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 
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Rough  Cost  Estimate:  Column  J _ 

49.  This  column  heading  and  information  in  the  column  cells  will  be  very  relevant 
in  the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

“ Provides  PM  with  information  on  the  cost  of  doing  the  HSI  at  each  level/activity  for  a 
program.  ” 

“Agree  with  the  spirit  of  what  this  is  attempting  to  accomplish.  I  suspect  that  a  cost 
estimate  is  not  necessary  for  every  single  sub-activity.  There  are  a  lot  of  government 
activities  that  are  taken  out  of  hide,  like  commenting  on  a  CDD  or  HP  writing  a  HSIP. 
Eventually,  you  probably  should  highlight  the  ones  you  especially  need  the  input  for  cost 
estimations.  Since  this  spreadsheet  lists  activities  to  be  completed  by  government  HSI 
WG,  then  the  cost  estimate  apparently  will  capture  costs  associated  government 
managemen  t  cost  portion  of  the  pie.  ” 

“Since  all  the  activities  have  such  variability  depending  on  the  program,  advocacy  of 
PM,  etc.  I’m  wondering  what  validity  the  estimate  will  have.  ” 


50.  This  column  heading  and  information  in  the  column  cells  will  be  very  useful  in 
the  HRL  framework  matrix  for  all  9  HRL  activity  listings. 

“Will  allow  PM  to  pick  and  choose  the  most  important  activities  based  on  budget 
restrictions.  ” 

“If  cost  is  an  ultimate  part  of  goal.  Possible  listing  in  effort  or  man-hours  versus  person 
days  would  be  relevant.  Depending  of  labor  mix,  cost  does  not  usually  occur  per  day 
(i.e.,  contract  personnel).  Also  might  be  good  Ideas  to  include  cost  drivers  or  cost 
estimation  relationship  columns  for  future  sourcing  and  appropriations.  ” 


51.  You  would  recommend  no  additional  column  topics  to  populate  the  matrix. 

“Relevant  to  Cost,  more  information  such  as  system,  and  drivers  may  be  important.  ROI 
Benefit  Analysis  and  Trade-off  columns  may  also  be  useful.  Common  language  as  used 
in  acquisition  or  reports  should  be  used  for  search  relationships  and  connections  ( I.e. 
target  document  is  good).  ” _ 
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INITIAL  DISTRIBUTION  LIST 


1.  Defense  Technical  Information  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 

3.  Paul  E.  O’Connor 

Naval  Postgraduate  School 
Monterey,  California 

4.  Michael  E.  McCauley 
Naval  Postgraduate  School 
Monterey,  California 

5.  Hector  M.  Acosta 
711  HPW 

Brooks  City-Base,  Texas 
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